Skip to content

Support kitty unicode placeholders (U=1) in addon-image - #6198

Open
valenvivaldi wants to merge 1 commit into
xtermjs:masterfrom
valenvivaldi:kitty-unicode-placeholders
Open

valenvivaldi wants to merge 1 commit into
xtermjs:masterfrom
valenvivaldi:kitty-unicode-placeholders

Conversation

@valenvivaldi

Copy link
Copy Markdown

Support kitty Unicode placeholders (U=1) in addon-image

Fixes #5711

Kitty's Unicode placeholders let a client create a virtual placement (a=T,U=1 / a=p,U=1), then show it by writing U+10EEEE cells. Because the image lives in ordinary text cells, it scrolls, reflows and gets erased together with the text. TUI apps rely on this to draw images inside their layout, and right now they fall back to text in xterm.js-based terminals.

What changes

  • Parsing: the new U key.
  • Virtual placements: U=1 decodes, crops and scales the image as before. It then stores the image as a virtual image in ImageStorage. Nothing is written to cells and the cursor does not move.
  • Rendering: ImageStorage.render looks for placeholder cells and draws the tile each one references.
    • The row/column diacritics come from kitty's rowcolumn-diacritics.txt.
    • The image id comes from the fg color (24-bit or 256-color). The optional third diacritic adds the most significant byte.
    • The placement id comes from the underline color.
    • Missing diacritics are inferred from the left neighbour, as the spec says.
    • Adjacent tiles are merged into one draw call, like regular tiles.
  • Hiding the glyph: after each parsed write (onWriteParsed), placeholder cells on the active screen are marked INVISIBLE. The text renderer then draws neither the placeholder glyph nor its diacritics. The cell shows its background, with the tile drawn on top.
  • Animation: retransmitting an image with a=T,U=1 and the same id swaps the virtual placement in place. The old entry is removed only after the new one is stored, so the image doesn't flicker or leak storage. Deletes (d=i/I, d=a/A), eviction and reset also clean up virtual placements.

Limitations

  • Placement ids: the buffer keeps an underline color only on underlined cells (ExtendedAttrs.isEmpty ignores it). Placement ids in the underline color therefore work only together with SGR 4. Without one, the cells use the most recent placement of the image. Clients that use a single placement per image, which is the common case, are unaffected. Fixing this would mean changing core, so I left it out of this PR.
  • Fitting: the image is stretched to c×r cells, the same as the existing direct placements. It is not aspect-fit and centred the way kitty does it.

Tests

  • Unit tests: KittyPlaceholder.test.ts covers diacritic decoding, including diacritics outside the BMP, 256-color ids, the MSB diacritic, underline placement ids, and inference from the left neighbour. Another test covers parsing of the U key.
  • Integration tests: a new Unicode placeholders (U=1) group in KittyGraphics.test.ts, with pixel checks on the image layer. It checks that:
    • the image is stored but not placed at the cursor;
    • each placeholder cell renders its own tile;
    • inferred columns work;
    • a single placeholder shows just its tile;
    • ids work through 256-color fg and underline placement;
    • retransmitting updates the cells in place;
    • deleting the image removes the placement;
    • glyphs are hidden;
    • unknown images draw nothing.
  • Results: the full addon-image suite passes on Chromium, Firefox and WebKit (665 passed). Lint is clean.
  • Manual check: in the demo I ran a script that sends what a TUI app sends: a 40×14 placeholder grid, with the image retransmitted at 20 fps under the same id. The image renders sharp and animates, with exactly one image kept in storage.

Virtual placements are stored without being written to cells and are
drawn wherever the client writes U+10EEEE placeholder cells: row/column
diacritics select the tile, the fg color (24-bit or 256, plus the
optional third diacritic) carries the image id and the underline color
the placement id. Missing diacritics are inferred from the left cell.

Placeholder cells are marked invisible after each parsed write so the
text renderer draws neither the glyph nor its diacritics. Retransmitting
an image with a=T,U=1 swaps the placement in place, so animations do
not flicker.

Fixes xtermjs#5711
@valenvivaldi

Copy link
Copy Markdown
Author

Heads-up: I saw #6145, #6132 and the feedback on #6098 about where kitty logic should live. This PR predates that redesign and puts the placeholder rendering in ImageStorage, so it likely doesn't fit the intended direction as-is. I'll leave it open in case it's useful as a reference: the diacritic decoder (KittyPlaceholder.ts) is independent of the storage model, and the integration tests describe the expected U=1 behaviour with pixel checks.

@petersindex

Copy link
Copy Markdown

Independent confirmation: I built the same feature separately (master...petersindex:xterm.js:feat/kitty-unicode-placeholders) and tested it end to end with Claude Code 2.1.290, whose plugin Image element draws through U=1 placeholders. Images render correctly in the placeholder cells. Hope this lands in whatever shape fits #6145.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Kitty graphics: Implement Unicode placeholder-based image display (U+10EEEE)

2 participants