Skip to content

Hands-free rewind: raise your hand to go back#25

Open
nathanael wants to merge 10 commits intof:masterfrom
nathanael:feat/hand-gesture-rewind
Open

Hands-free rewind: raise your hand to go back#25
nathanael wants to merge 10 commits intof:masterfrom
nathanael:feat/hand-gesture-rewind

Conversation

@nathanael
Copy link
Copy Markdown

Why

When you're mid-presentation and stumble over a line, your hands are often gesturing or holding notes — you can't easily reach the keyboard to rewind. This feature lets you raise your hand to pause and automatically rewind the teleprompter, then lower it to resume exactly where you left off. No keyboard, no mouse, no breaking your flow.

What it does

  • Raise hand → teleprompter pauses and starts rewinding through words
  • Hand height controls speed — higher hand = faster rewind
  • Lower hand → speech recognition resumes from the new position
  • Visual indicator — spinning circle in the corner shows rewind is active
  • Uses Vision framework for real-time hand pose detection via built-in camera
  • Hysteresis prevents jittery toggling near the detection threshold

How it works

  • HandGestureController captures camera frames and runs VNDetectHumanHandPoseRequest
  • Tracks wrist Y position with a smoothing window for stable detection
  • SpeechRecognizer gains pauseForRewind(), rewindByWords(), and resumeAfterRewind()
  • Rewind speed scales with hand height (1/2/4 words per tick)
  • Camera session properly cleaned up on stop to prevent resource leaks

Test plan

  • Start teleprompter in word tracking mode, raise hand — verify pause + rewind
  • Vary hand height — verify rewind speed changes
  • Lower hand — verify speech recognition resumes
  • Verify spinning indicator appears/disappears correctly
  • Test with different lighting conditions
  • Verify camera permission prompt appears on first use

Nathanael and others added 10 commits March 22, 2026 20:04
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Address mode-specific rewind mechanisms: recognizedCharCount for
wordTracking, timerWordProgress for classic/silencePaused. Add edge
cases for camera permission, multiple hands, no camera hardware.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix silencePaused mode grouping, correct method names to
show()/dismiss()/forceClose(), add .onDisappear cleanup,
fix O(n) string indexing in rewindByWords.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Start/stop camera with reading sessions. Both NotchOverlayView
and FloatingOverlayView observe HandGestureController to drive
rewind in all three listening modes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
AVCaptureDALDevice does not support setActiveVideoMinFrameDuration.
The .low session preset already limits resource usage.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Raise hand to rewind script, lower to resume. Green spinning
circle appears when rewinding. Uses Vision framework hand pose
detection with hysteresis thresholds to prevent flicker.

Fixes: threshold tuning, camera DAL crash, indicator z-order,
callback-based state change, session cleanup on stop.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a discoverable toggle with description explaining the feature:
raise hand to pause and rewind, height controls speed, lower to resume.
Enabled by default. When disabled, camera is not started and no hand
detection runs. Prevents surprise camera access for users who do not
want it.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant