feat: render inline images via Kitty graphics protocol#77
Open
trieloff wants to merge 3 commits into
Open
Conversation
Intercepts APC `\x1b_G...\x1b\\` sequences from input streams in
`@wterm/dom`, decodes the base64 PNG payload (including chunked
`m=1`/`m=0` transfers), and renders the image as an absolutely-
positioned `<img>` overlay inside the cell grid. The overlay layer
stays aligned with its anchor row across scrollback growth because
the renderer inserts new scrollback rows above existing grid rows,
keeping content's pixel position invariant.
Supports actions `t`/`T`/`p`/`d` with PNG format (`f=100`) over the
direct base64 transport, plus `c=`/`r=` cell-fit sizing, `i=`/`I=`
identification, and the `C=1` no-cursor-movement opt-out. The
default `T`/`p` cursor advance is implemented by writing newlines
to the core based on the image's row count (taken from `r=` or
parsed from the PNG IHDR).
Not in scope: raw RGB/RGBA frames, file/shared-memory transports,
virtual placement via Unicode placeholders, animations, Sixel, and
iTerm2 inline images.
Opt out with `new WTerm(el, { images: false })`. The same option
is threaded through `@wterm/react` and `@wterm/vue`.
Closes vercel-labs#60
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Lars Trieloff <lars@trieloff.net>
|
@claude is attempting to deploy a commit to the Vercel Labs Team on Vercel. A member of the Team first needs to authorize it. |
trieloff
commented
May 21, 2026
Author
trieloff
left a comment
trieloff
left a comment
There was a problem hiding this comment.
Don't reformat lines that you didn't touch.
Restores the original compact formatting on lines the Kitty graphics feature does not need to touch, addressing PR vercel-labs#77 review feedback: - packages/@wterm/dom/src/terminal.css: --term-font-family back to a single line with single quotes; @Keyframes cursor-blink back to 2-line compact form. Keeps position: relative on .term-grid plus the new .term-image-layer / .term-image rules. - packages/@wterm/dom/README.md: Options and Methods tables restored to compact pipe-table style; new images row added in the same style; Inline images section preserved. - apps/docs/src/app/api-reference/page.mdx: pre-existing <td><code> cells back to one line; new images row added in the same compact JSX style. - README.md: Packages table restored to compact form; Inline images Features bullet preserved.
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds inline PNG image rendering support via the Kitty terminal graphics protocol across DOM/React/Vue packages, with docs and an example app.
Changes:
- Introduces a streaming Kitty APC parser and an
<img>overlay layer in@wterm/dom, wired intoWTerm.write(). - Adds an
imagesoption/prop (defaulttrue) to Dom/React/Vue and updates docs/API reference accordingly. - Adds tests for the Kitty parser and an example Vite app demonstrating chunked image transfer.
Reviewed changes
Copilot reviewed 21 out of 23 changed files in this pull request and generated 14 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/@wterm/vue/src/Terminal.ts | Adds images prop and forwards it into WTerm options. |
| packages/@wterm/react/src/Terminal.tsx | Adds images prop and forwards it into WTerm options. |
| packages/@wterm/dom/src/wterm.ts | Wires Kitty filtering + overlay into write path; cursor advance for images; cleanup on destroy. |
| packages/@wterm/dom/src/terminal.css | Adds overlay/image layer styling and makes grid position: relative. |
| packages/@wterm/dom/src/renderer.ts | Preserves non-row children (e.g., image overlay) across renderer setup. |
| packages/@wterm/dom/src/png.ts | Adds lightweight PNG IHDR dimension extraction helper. |
| packages/@wterm/dom/src/kitty-graphics.ts | Adds streaming Kitty graphics APC parser/filter. |
| packages/@wterm/dom/src/index.ts | Exports new Kitty graphics and overlay APIs from @wterm/dom. |
| packages/@wterm/dom/src/image-overlay.ts | Adds <img> overlay implementation for rendering placements. |
| packages/@wterm/dom/src/tests/wterm.test.ts | Updates write-path expectations + adds coverage for images: false. |
| packages/@wterm/dom/src/tests/kitty-graphics.test.ts | Adds unit tests for the Kitty streaming filter, chunking, reset. |
| packages/@wterm/dom/README.md | Documents the new images option and usage example. |
| examples/kitty-images/wterm-dom.d.ts | Declares CSS module import for the example. |
| examples/kitty-images/vite.config.ts | Vite config for the new example app. |
| examples/kitty-images/tsconfig.json | TypeScript configuration for the example app. |
| examples/kitty-images/src/main.ts | Demo app that generates a PNG and sends chunked Kitty APC sequences. |
| examples/kitty-images/package.json | Example app package metadata and scripts. |
| examples/kitty-images/index.html | HTML entry for the example app. |
| examples/kitty-images/README.md | Example app instructions and explanation. |
| apps/docs/src/app/configuration/page.mdx | Adds images to shared options + new inline images section. |
| apps/docs/src/app/api-reference/page.mdx | Adds images option to the API reference table. |
| README.md | Mentions inline images as a top-level feature. |
Files not reviewed (1)
- pnpm-lock.yaml: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+92
to
+103
| export class KittyGraphicsFilter { | ||
| private state: State = State.Idle; | ||
| /** Buffered Kitty APC payload (bytes between `\x1b_G` and the terminator). */ | ||
| private apcBuf: number[] = []; | ||
| /** Pending chunked transfers keyed by image id (i=) or image number (-I=). */ | ||
| private pendingChunks = new Map<string, PendingChunk>(); | ||
|
|
||
| /** | ||
| * Push a chunk of bytes through the filter. Returns the ordered list of | ||
| * pass-through text segments and completed graphics events. | ||
| */ | ||
| push(input: Uint8Array): StreamEvent[] { |
| this.pendingChunks.clear(); | ||
| } | ||
|
|
||
| private _completeApc(events: StreamEvent[]): void { |
Comment on lines
+220
to
+221
| const control = parseControl(decodeAscii(ctrlBytes)); | ||
| const payloadB64 = decodeAscii(payloadBytes); |
| const more = control.m === 1; | ||
| const key = chunkKey(control); | ||
|
|
||
| if (more || this.pendingChunks.has(key)) { |
Comment on lines
+249
to
+263
| data: decodeBase64(completed.payload), | ||
| }, | ||
| }); | ||
| return; | ||
| } | ||
|
|
||
| events.push({ | ||
| type: "graphics", | ||
| event: { | ||
| control, | ||
| data: decodeBase64(payloadB64), | ||
| }, | ||
| }); | ||
| } | ||
| } |
Comment on lines
+121
to
+125
| const blob = new Blob([new Uint8Array(stored.data)], { type: "image/png" }); | ||
| img.src = URL.createObjectURL(blob); | ||
| img.addEventListener("load", () => URL.revokeObjectURL(img.src), { | ||
| once: true, | ||
| }); |
Comment on lines
+188
to
+192
| function imageKey(control: KittyControl): string | null { | ||
| if (typeof control.i === "number") return `i:${control.i}`; | ||
| if (typeof control.I === "number") return `I:${control.I}`; | ||
| return "i:0"; | ||
| } |
| When `images: true` (default), wterm intercepts Kitty graphics protocol APC sequences and renders the transmitted PNG as an absolutely-positioned `<img>` overlay aligned to the cell grid. Supports inline base64 transfers (`f=100`) and multi-chunk `m=1`/`m=0` payloads. Actions: `t` (transmit), `T` (transmit + display), `p` (put placement), `d` (delete). | ||
|
|
||
| ```ts | ||
| const png = await fetch("/icon.png").then((r) => r.bytes()); |
| const encoded = new TextEncoder().encode("hello"); | ||
| expect(mockBridge.writeRaw).toHaveBeenCalledWith(encoded); | ||
| }); | ||
|
|
Comment on lines
+178
to
+183
| it("falls back to writeString when images are disabled", async () => { | ||
| const term = new WTerm(element, { autoResize: false, images: false }); | ||
| await term.init(); | ||
| term.write("hello"); | ||
| expect(mockBridge.writeString).toHaveBeenCalledWith("hello"); | ||
| }); |
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
…test Addresses automated review feedback on PR vercel-labs#77 (Copilot + Vercel VADE): image-overlay.ts: - Track objectUrl per Placement; revoke on replace, _delete, and clear - Add error listener alongside load to revoke on decode failures - imageKey() now returns string (always); remove three dead null guards kitty-graphics.ts: - Wrap decodeBase64 in try/catch; silently drop malformed APCs - Cap pendingChunks at MAX_PENDING_CHUNKS=8 and MAX_PENDING_BASE64_BYTES=32MiB with oldest-entry eviction; track running pendingBytes - Replace per-byte decodeAscii loop with TextDecoder('latin1') dom/README.md: - Use new Uint8Array(await r.arrayBuffer()) in the inline-images snippet instead of non-portable Response.bytes() Tests: - wterm.test.ts: new integration test asserts the bridge never sees ESC bytes when images are enabled and a Kitty APC is in the input stream - kitty-graphics.test.ts: cover invalid-base64 silent drop and chunk-count cap eviction 84/84 @wterm/dom tests pass; pnpm -r type-check clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #60.
Adds inline image rendering to
wtermvia the Kitty terminal graphics protocol. Everything lives in@wterm/dom— no changes to the Zig core or libghostty.@wterm/domintercepts APC\x1b_G...\x1b\\sequences before they reach the VT core, parses thekey=valuecontrol + base64 payload (with chunkedm=1/m=0accumulation), and emits image events.<img>inside theterm-gridcontainer. Because the existing renderer inserts new scrollback rows above the grid, content's pixel offset is invariant under scroll-up — so an image stays pinned to the cells where it was placed, with no per-row anchor bookkeeping.T/pcursor advancement is implemented by writing newlines to the core based on either ther=control or the PNG IHDR dimensions (via a tiny header reader).new WTerm(el, { images: false }). The same flag is threaded through@wterm/reactand@wterm/vue.Scope (intentional MVP)
In:
t(transmit),T(transmit + display),p(put placement),d(delete)f=100)t=d, default), including chunkedm=1/m=0i=/I=identifiers,p=placement IDs,c=/r=cell sizing,z=z-index,X=/Y=cell offsets,C=1cursor-stay opt-outOut (follow-ups):
f=24/f=32)t=f/t=s/t=t)Each scope cut is called out in the docs so users know what to expect.
What changed
packages/@wterm/dom/src/kitty-graphics.ts(new) — streaming APC parser/filterpackages/@wterm/dom/src/image-overlay.ts(new) — DOM overlay with store / place / deletepackages/@wterm/dom/src/png.ts(new) — PNG IHDR dimension readerpackages/@wterm/dom/src/wterm.ts— wired filter intowrite(), addedimagesoptionpackages/@wterm/dom/src/renderer.ts—setup()now preserves non-row children so the overlay survivesResizeObserver-driven resizespackages/@wterm/dom/src/terminal.css—.term-grid { position: relative }+.term-image-layer/.term-imagerulespackages/@wterm/{react,vue}/src/Terminal.*— pass-throughimagespropexamples/kitty-images/(new) — self-contained Vite demo that draws a PNG on a<canvas>and transmits it via a chunked APCREADME,@wterm/domREADME,apps/docs/src/app/configuration/page.mdx,apps/docs/src/app/api-reference/page.mdxTest plan
@wterm/dom(12 new tests inkitty-graphics.test.tscovering plain text passthrough, CSI passthrough, non-Kitty APC passthrough, single sequence, BEL terminator, chunked split-across-writes, byte-by-byte streaming, delete action, ESC followed by non-_,_followed by non-G, text-run coalescing, and reset).pnpm -r type-check)@wterm/core,@wterm/react,@wterm/vue,@wterm/just-bash,@wterm/markdown/tmp/wterm-kitty.png.How to demo
pnpm install pnpm --filter kitty-images-example dev # then open the URL portless assigns (or run `vite preview` if you don't have portless)A generated 200×100 PNG (gradient + "wterm 🚀" text) is transmitted via the protocol's chunked APC transport.