Skip to content

Add floating playback controls for agent speech - #1216

Open
pineapplestrikesback wants to merge 1 commit into
jamiepine:mainfrom
pineapplestrikesback:playback-controls
Open

pineapplestrikesback wants to merge 1 commit into
jamiepine:mainfrom
pineapplestrikesback:playback-controls

Conversation

@pineapplestrikesback

Copy link
Copy Markdown

Agent speech currently plays through a floating status pill without a way to pause, seek, change speed, or replay it there. This adds a compact player using the existing completed recording: pause/resume, position and duration, seeking, replay, stop, and a saved 0.75×–2× speed for agent speech.

The same widget mirrors the main waveform player and sends commands back to its existing audio source. Selecting a recording alone does not open the widget. Finished playback stays visible for five seconds for replay; History keeps the recording afterward.

Implementation

  • Extract completed-file speech playback into a testable transport with media-clock progress, pitch-preserving browser speed, and cleanup of stale generation events and play promises.
  • Retain the existing latest-speech-wins behavior and backend endpoints. No model, generation, streaming, Inbox, or attention-tracking changes.
  • Keep the main player's existing WebAudio backend and normal speed. Native device routing exposes its existing limitations: no seeking/speed, and stop/start restarts playback.
  • Serialize native playback starts/stops so cancelling an in-flight start cannot leave audio running or stop a newer recording. Pause the current waveform before awaiting native cleanup.
  • Make space for the controls in the floating window; show it without activation on macOS and restore the preceding app only when the widget still owns focus. Preserve existing application-close and server behavior.
  • Translate controls into all nine existing locales, add a usage/manual-test guide, and run playback tests in frontend CI. No new dependencies.

Validation

  • 30 playback tests passed, 507 assertions: transport lifecycle, queued media-event races, native start/stop ordering, current-session widget commands, persisted speed, rendered controls, and locale parity.
  • App/web TypeScript checks passed. Web production build and Tauri frontend production build passed.
  • Targeted Biome checks passed for new playback modules, tests, and DictateWindow. Existing unrelated AudioPlayer lint diagnostics remain.
  • macOS Rust cargo check passed with a temporary bundle.externalBin=[] override because the checkout has no built Python sidecar. Upstream's stale root-package version in Cargo.lock was resolved temporarily for the check and restored; no lockfile changes are included. This was a compile check, not a packaged build.
  • An isolated Brave fixture exercised the real component and transport with a muted five-minute WAV: pause, keyboard seek, speed change, saved speed after reload, completion, and replay using the same audio instance. Observed seek/speed handlers took 0.1–0.3 ms in this fixture. This does not measure audible speech quality or native WKWebView behavior.
  • Packaged macOS 27.0.1 arm64 pass: an isolated, signed release app used the branch backend on a separate port with seeded speech recordings. Native pause/resume, pointer seeking, speed, persisted speed after process restart, completion/replay, stop, main-player mirroring and replacement requests passed. A manual click focused the player; dismissal returned focus to the prior app, confirmed by the tester and a passive OS activation log. Playback/focus code matched this patch byte-for-byte. No additional code fixes were required.

The macOS package test used a separately supervised Python backend and bypassed synthesis only. It does not validate Python-sidecar bundling, installer/notarization/updater delivery, model inference, selected-device hardware routing, acoustic output quality, or Windows/Linux. Native keyboard seeking was not established by background automation. The manual checklist is in docs/PLAYBACK_CONTROLS.md.

macOS preview

Packaged macOS player paused after seeking, at 1.5× speed

@pineapplestrikesback
pineapplestrikesback marked this pull request as ready for review October 22, 2026 12:27

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant