Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .changes/native-desktop-recording.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
"tauri-plugin-camera-kessdev": "minor"
"tauri-plugin-camera-kessdev-js": "minor"
---

Add an opt-in native desktop camera backend (Cargo feature `native-desktop`,
cross-platform GStreamer); it owns the camera and captures once, fanning out to a
live preview (JPEG frames delivered to the frontend) and a native encoder, so the UI
is not blocked and `stopRecording()` does not freeze. Expose `startPreview({ onFrame })`,
the `PreviewFrame` type and `getCapabilities().supportsNativeRecording`.
10 changes: 9 additions & 1 deletion .github/workflows/verify.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,9 @@ jobs:
librsvg2-dev \
libxdo-dev \
libssl-dev \
build-essential
build-essential \
libgstreamer1.0-dev \
libgstreamer-plugins-base1.0-dev

- name: format
run: cargo fmt --all -- --check
Expand All @@ -37,6 +39,12 @@ jobs:
- name: test
run: cargo test

- name: clippy (native desktop)
run: cargo clippy --features native-desktop --all-targets -- -D warnings

- name: test (native desktop)
run: cargo test --features native-desktop

- name: check mobile target
run: cargo check --target aarch64-linux-android

Expand Down
10 changes: 10 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,21 @@ rust-version = "1.77.2"
exclude = ["/examples", "/dist-js", "/guest-js", "/node_modules", "/.changes", "/.github"]
links = "tauri-plugin-camera-kessdev"

[features]
default = []
# Native desktop camera backend (capture -> preview frames + native encoder).
# Off by default to avoid requiring the platform dev libraries (e.g. GStreamer
# on Linux) for every build.
native-desktop = ["dep:gstreamer", "dep:gstreamer-app", "dep:base64"]

[dependencies]
tauri = { version = "2.11.3" }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
thiserror = "2"
base64 = { version = "0.22", optional = true }
gstreamer = { version = "0.23", optional = true }
gstreamer-app = { version = "0.23", optional = true }

[target.'cfg(target_os = "windows")'.dependencies]
webview2-com = "0.39"
Expand Down
34 changes: 33 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,36 @@ preview's video track (only the microphone is acquired separately), so the
`<video>` element bound to `getPreviewStream()` keeps playing. The recording
mime type is chosen from the browser's supported types (WebM or MP4).

### Desktop native backend (feature `native-desktop`)

By default the desktop camera runs in the webview (`getUserMedia` / `MediaRecorder`).
Compiling the plugin with the optional Cargo feature `native-desktop` (GStreamer,
cross-platform) makes the plugin **own the camera natively**: it captures once and fans out to a
**live preview** (JPEG frames delivered to your app) and a **native encoder** — so the
UI is never blocked and `stopRecording()` does not freeze the window (the WebKitGTK
problem). Query availability with `getCapabilities().supportsNativeRecording`.

```ts
import type { Channel } from '@tauri-apps/api/core'
import { startPreview, startRecording, stopRecording, type PreviewFrame } from 'tauri-plugin-camera-kessdev'

const frames = new Channel<PreviewFrame>()
frames.onmessage = ({ data }) => drawToCanvas(data) // base64 JPEG
await startPreview({ onFrame: frames })
await startRecording()
// ... user records ...
const video = await stopRecording() // { path, duration }
```

- Enabling the feature requires GStreamer dev libraries for your platform (on Linux,
`libgstreamer1.0-dev` and `libgstreamer-plugins-base1.0-dev`) plus runtime plugins
for the source (`pipewiresrc`/`v4l2src`, `mfvideosrc`/`ksvideosrc`, `avfvideosrc`),
`x264enc`, `mp4mux` and AAC; on Windows/macOS install the GStreamer distribution.
- With the native backend active, `getPreviewStream()` returns `null` (there is no
webview `MediaStream`); render the frames in a `<canvas>` instead.
- `capture()` returns the most recent native preview frame as a base64 data URL
(`Photo.dataUrl` / `Photo.thumbnail`).

### Video thumbnail

Pass `{ thumbnail: true }` to `stopRecording()` to include a base64-encoded JPEG
Expand Down Expand Up @@ -504,7 +534,8 @@ type PhotoFormat = 'jpeg' | 'png' | 'heif'
type VideoQuality = 'sd' | 'hd' | 'fhd' | 'uhd'
type PermissionState = 'granted' | 'denied' | 'prompt' // re-exported from @tauri-apps/api/core

interface StartPreviewOptions { camera?: CameraDirection; cameraId?: string; windowed?: boolean }
interface StartPreviewOptions { camera?: CameraDirection; cameraId?: string; windowed?: boolean; onFrame?: Channel<PreviewFrame> }
interface PreviewFrame { data: string; width: number; height: number }
interface CaptureOptions {
flash?: FlashMode
resolution?: Resolution
Expand Down Expand Up @@ -555,6 +586,7 @@ interface CameraCapabilities {
exposureModes?: ExposureMode[]
whiteBalanceModes?: WhiteBalanceMode[]
supportsWhiteBalanceTemperature?: boolean
supportsNativeRecording?: boolean
photoResolutions?: Resolution[]
photoAspects?: string[]
videoQualities?: VideoQuality[]
Expand Down
6 changes: 6 additions & 0 deletions build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,12 @@ const COMMANDS: &[&str] = &[
"request_permissions",
"check_permissions",
"open_app_settings",
"native_desktop_available",
"native_start_preview",
"native_stop_preview",
"native_start_recording",
"native_stop_recording",
"native_capture",
];

fn main() {
Expand Down
56 changes: 56 additions & 0 deletions guest-js/index.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,14 @@ describe('desktop introspection', () => {
expect(caps.focusModes).toEqual(['continuous', 'locked'])
})

it('reports native backend availability in capabilities', async () => {
await stopPreview()
invokeMock.mockResolvedValue(true)
const caps = await getCapabilities()
expect(invokeMock).toHaveBeenCalledWith('plugin:camera-kessdev|native_desktop_available')
expect(caps.supportsNativeRecording).toBe(true)
})

it('startPreview selects a camera by deviceId on desktop', async () => {
const track = { stop: vi.fn() }
const stream = { getTracks: () => [track], getVideoTracks: () => [track] }
Expand Down Expand Up @@ -524,3 +532,51 @@ describe('desktop capture config', () => {
await expect(capture({ format: 'heif' })).rejects.toThrow(/not supported on desktop/)
})
})

describe('desktop native backend', () => {
beforeEach(() => {
setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64)')
})

afterEach(async () => {
await stopPreview()
vi.unstubAllGlobals()
vi.restoreAllMocks()
})

it('startPreview with onFrame uses the native command when available', async () => {
invokeMock.mockResolvedValueOnce(true) // native_desktop_available
invokeMock.mockResolvedValueOnce(undefined) // native_start_preview
const channel = { onmessage: vi.fn() } as unknown as Channel<unknown>

await startPreview({ onFrame: channel as never })

expect(invokeMock).toHaveBeenNthCalledWith(1, 'plugin:camera-kessdev|native_desktop_available')
expect(invokeMock).toHaveBeenNthCalledWith(2, 'plugin:camera-kessdev|native_start_preview', {
onFrame: channel
})
})

it('startRecording/stopRecording use the native commands when available', async () => {
invokeMock.mockResolvedValueOnce(true) // native_desktop_available
invokeMock.mockResolvedValueOnce('/tmp/rec.mp4') // native_start_recording
await startRecording()
expect(invokeMock).toHaveBeenNthCalledWith(2, 'plugin:camera-kessdev|native_start_recording', {})

invokeMock.mockResolvedValueOnce(1.5) // native_stop_recording
const video = await stopRecording()
expect(invokeMock).toHaveBeenCalledWith('plugin:camera-kessdev|native_stop_recording')
expect(video.path).toBe('/tmp/rec.mp4')
expect(video.duration).toBe(1.5)
})

it('capture uses the native frame when available', async () => {
invokeMock.mockResolvedValueOnce(true) // native_desktop_available
invokeMock.mockResolvedValueOnce({ data: 'AAAA', width: 640, height: 480 }) // native_capture
const photo = await capture()
expect(invokeMock).toHaveBeenNthCalledWith(2, 'plugin:camera-kessdev|native_capture')
expect(photo.dataUrl).toBe('data:image/jpeg;base64,AAAA')
expect(photo.width).toBe(640)
expect(photo.height).toBe(480)
})
})
66 changes: 64 additions & 2 deletions guest-js/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,24 @@ export { convertFileSrc }
export type CameraDirection = 'front' | 'back'
export type FlashMode = 'on' | 'off' | 'auto'

/** A preview frame delivered by the native desktop backend. */
export interface PreviewFrame {
/** Base64-encoded JPEG image data. */
data: string
width: number
height: number
}

export interface StartPreviewOptions {
camera?: CameraDirection
/** Specific camera/lens id from `getAvailableCameras()`. Takes precedence over `camera`. */
cameraId?: string
windowed?: boolean
/**
* Desktop native backend only: receives live preview frames to render in a
* `<canvas>`. Ignored by the webview fallback and on mobile.
*/
onFrame?: Channel<PreviewFrame>
}

export type PhotoFormat = 'jpeg' | 'png' | 'heif'
Expand Down Expand Up @@ -137,6 +150,8 @@ export interface CameraCapabilities {
whiteBalanceModes?: WhiteBalanceMode[]
/** Whether an arbitrary color temperature (Kelvin) can be applied. */
supportsWhiteBalanceTemperature?: boolean
/** Whether the native desktop camera backend is available (desktop). */
supportsNativeRecording?: boolean
photoResolutions?: Resolution[]
photoAspects?: string[]
videoQualities?: VideoQuality[]
Expand Down Expand Up @@ -200,6 +215,7 @@ let activeStream: MediaStream | null = null
let activeVideo: HTMLVideoElement | null = null
let mediaRecorder: MediaRecorder | null = null
let recordingAudioStream: MediaStream | null = null
let nativeRecordingPath: string | null = null
let recordedChunks: Blob[] = []
let recordedBytes = 0
let recordingStartedAt = 0
Expand Down Expand Up @@ -382,6 +398,10 @@ export async function startPreview(options?: StartPreviewOptions): Promise<void>
await invoke('plugin:camera-kessdev|start_preview', { ...options })
return
}
if (options?.onFrame && (await nativeDesktopAvailable())) {
await invoke('plugin:camera-kessdev|native_start_preview', { onFrame: options.onFrame })
return
}
await ensureDesktopStream(options)
}

Expand All @@ -391,15 +411,21 @@ export async function stopPreview(): Promise<void> {
await invoke('plugin:camera-kessdev|stop_preview')
return
}
if (await nativeDesktopAvailable()) {
await invoke('plugin:camera-kessdev|native_stop_preview')
}
teardownDesktopStream()
}

/**
* Returns the active desktop preview stream so the application can render it in
* a `<video>` element. Returns `null` on mobile (the native preview is used).
* a `<video>` element. Returns `null` on mobile, and on desktop when the native
* camera backend is active (use `startPreview({ onFrame })` + a `<canvas>`
* instead).
*/
export async function getPreviewStream(): Promise<MediaStream | null> {
if (isMobile()) return null
if (await nativeDesktopAvailable()) return null
return activeStream
}

Expand All @@ -418,6 +444,21 @@ export async function capture(options?: CaptureOptions): Promise<Photo> {
if (isMobile()) {
return await invoke<Photo>('plugin:camera-kessdev|capture', { ...options })
}
if (await nativeDesktopAvailable()) {
const frame = await invoke<PreviewFrame | null>('plugin:camera-kessdev|native_capture')
if (!frame) {
throw new Error('No preview frame available. Call startPreview({ onFrame }) first.')
}
const dataUrl = `data:image/jpeg;base64,${frame.data}`
return {
dataUrl,
width: frame.width,
height: frame.height,
orientation: 1,
format: 'jpeg',
thumbnail: dataUrl
}
}
return captureDesktopFrame(options)
}

Expand Down Expand Up @@ -538,6 +579,13 @@ export async function getAvailableCameras(): Promise<CameraInfo[]> {
return enumerateDesktopCameras()
}

/** Whether the native desktop camera backend is available for this build. */
function nativeDesktopAvailable(): Promise<boolean> {
return Promise.resolve(invoke<boolean>('plugin:camera-kessdev|native_desktop_available'))
.then((value) => value === true)
.catch(() => false)
}

/**
* Returns the declarative capabilities of a camera.
*
Expand All @@ -550,7 +598,9 @@ export async function getCapabilities(cameraId?: string): Promise<CameraCapabili
if (isMobile()) {
return await invoke<CameraCapabilities>('plugin:camera-kessdev|get_capabilities', { cameraId })
}
return desktopCapabilities(cameraId)
const caps = desktopCapabilities(cameraId)
caps.supportsNativeRecording = await nativeDesktopAvailable()
return caps
}

// ---------------------------------------------------------------------------
Expand Down Expand Up @@ -622,6 +672,11 @@ export async function startRecording(options?: StartRecordingOptions): Promise<v
return
}

if (await nativeDesktopAvailable()) {
nativeRecordingPath = await invoke<string>('plugin:camera-kessdev|native_start_recording', {})
return
}

// Reuse the active preview video track when possible so the preview is not
// invalidated (which would freeze the app's <video> element).
let videoTrack = activeStream?.getVideoTracks()[0]
Expand Down Expand Up @@ -691,6 +746,13 @@ export async function stopRecording(options?: StopRecordingOptions): Promise<Vid
return await invoke<Video>('plugin:camera-kessdev|stop_recording', { ...options })
}

if (nativeRecordingPath) {
const path = nativeRecordingPath
nativeRecordingPath = null
const duration = await invoke<number>('plugin:camera-kessdev|native_stop_recording')
return { path, duration }
}

if (!mediaRecorder) {
throw new Error('Recording is not active. Call startRecording() first.')
}
Expand Down
13 changes: 13 additions & 0 deletions permissions/autogenerated/commands/native_capture.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Automatically generated - DO NOT EDIT!

"$schema" = "../../schemas/schema.json"

[[permission]]
identifier = "allow-native-capture"
description = "Enables the native_capture command without any pre-configured scope."
commands.allow = ["native_capture"]

[[permission]]
identifier = "deny-native-capture"
description = "Denies the native_capture command without any pre-configured scope."
commands.deny = ["native_capture"]
13 changes: 13 additions & 0 deletions permissions/autogenerated/commands/native_desktop_available.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Automatically generated - DO NOT EDIT!

"$schema" = "../../schemas/schema.json"

[[permission]]
identifier = "allow-native-desktop-available"
description = "Enables the native_desktop_available command without any pre-configured scope."
commands.allow = ["native_desktop_available"]

[[permission]]
identifier = "deny-native-desktop-available"
description = "Denies the native_desktop_available command without any pre-configured scope."
commands.deny = ["native_desktop_available"]
13 changes: 13 additions & 0 deletions permissions/autogenerated/commands/native_start_preview.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Automatically generated - DO NOT EDIT!

"$schema" = "../../schemas/schema.json"

[[permission]]
identifier = "allow-native-start-preview"
description = "Enables the native_start_preview command without any pre-configured scope."
commands.allow = ["native_start_preview"]

[[permission]]
identifier = "deny-native-start-preview"
description = "Denies the native_start_preview command without any pre-configured scope."
commands.deny = ["native_start_preview"]
13 changes: 13 additions & 0 deletions permissions/autogenerated/commands/native_start_recording.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Automatically generated - DO NOT EDIT!

"$schema" = "../../schemas/schema.json"

[[permission]]
identifier = "allow-native-start-recording"
description = "Enables the native_start_recording command without any pre-configured scope."
commands.allow = ["native_start_recording"]

[[permission]]
identifier = "deny-native-start-recording"
description = "Denies the native_start_recording command without any pre-configured scope."
commands.deny = ["native_start_recording"]
Loading
Loading