Skip to content

feat: cursor follow crop with text cursor focus mode #640

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
walters954 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: cursor follow crop with text cursor focus mode #640

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
536 changes: 417 additions & 119 deletions src/components/video-editor/CropControl.tsx
View file
Open in desktop

Large diffs are not rendered by default.

18 changes: 18 additions & 0 deletions src/components/video-editor/VideoEditor.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,7 @@ import {
type ClipRegion,
type CropRegion,
type CursorClickEffectStyle,
type CursorFollowCropSettings,
type CursorStyle,
type CursorTelemetryPoint,
clampFocusToDepth,
Expand All @@ -187,6 +188,7 @@ import {
DEFAULT_CONNECTED_ZOOM_EASING,
DEFAULT_CONNECTED_ZOOM_GAP_MS,
DEFAULT_CROP_REGION,
DEFAULT_CURSOR_FOLLOW_CROP,
DEFAULT_CURSOR_STYLE,
DEFAULT_FIGURE_DATA,
DEFAULT_WEBCAM_OVERLAY,
Expand Down Expand Up @@ -489,6 +491,9 @@ export default function VideoEditor() {
const [padding, setPadding] = useState(initialEditorPreferences.padding);
const [frame, setFrame] = useState<string | null>(initialEditorPreferences.frame);
const [cropRegion, setCropRegion] = useState<CropRegion>(DEFAULT_CROP_REGION);
const [cursorFollowCrop, setCursorFollowCrop] = useState<CursorFollowCropSettings>(
DEFAULT_CURSOR_FOLLOW_CROP,
);
const [webcam, setWebcam] = useState<WebcamOverlaySettings>(
initialEditorPreferences.webcam ?? DEFAULT_WEBCAM_OVERLAY,
);
Expand Down Expand Up @@ -1095,6 +1100,7 @@ export default function VideoEditor() {
borderRadius,
padding,
cropRegion,
cursorFollowCrop,
webcam,
webcamUrl:
resolvedWebcamVideoUrl ??
Expand Down Expand Up @@ -1629,6 +1635,7 @@ export default function VideoEditor() {
padding: Padding;
frame: string | null;
cropRegion: CropRegion;
cursorFollowCrop: CursorFollowCropSettings;
webcam: WebcamOverlaySettings;
zoomRegions: ZoomRegion[];
trimRegions: TrimRegion[];
Expand Down Expand Up @@ -1737,6 +1744,7 @@ export default function VideoEditor() {
padding,
frame,
cropRegion,
cursorFollowCrop,
webcam,
zoomRegions,
trimRegions,
Expand Down Expand Up @@ -1803,6 +1811,7 @@ export default function VideoEditor() {
borderRadius,
padding,
cropRegion,
cursorFollowCrop,
webcam,
zoomRegions,
trimRegions,
Expand Down Expand Up @@ -1993,6 +2002,7 @@ export default function VideoEditor() {
setPadding(normalizedEditor.padding);
setFrame(normalizedEditor.frame);
setCropRegion(normalizedEditor.cropRegion);
setCursorFollowCrop(normalizedEditor.cursorFollowCrop);
setWebcam(normalizedEditor.webcam);
setZoomRegions(normalizedEditor.zoomRegions);
setTrimRegions(normalizedEditor.trimRegions);
Expand Down Expand Up @@ -4220,6 +4230,7 @@ export default function VideoEditor() {
padding,
videoPadding: padding,
cropRegion,
cursorFollowCrop,
webcam,
webcamUrl:
resolvedWebcamVideoUrl ??
Expand Down Expand Up @@ -4403,6 +4414,7 @@ export default function VideoEditor() {
borderRadius,
padding,
cropRegion,
cursorFollowCrop,
webcam,
webcamUrl:
resolvedWebcamVideoUrl ??
Expand Down Expand Up @@ -4704,6 +4716,7 @@ export default function VideoEditor() {
borderRadius,
padding,
cropRegion,
cursorFollowCrop,
webcam,
resolvedWebcamVideoUrl,
annotationRegions,
Expand Down Expand Up @@ -5118,6 +5131,7 @@ export default function VideoEditor() {
padding={padding}
frame={frame}
cropRegion={cropRegion}
cursorFollowCrop={cursorFollowCrop}
webcam={webcam}
webcamVideoPath={webcam.sourcePath ? resolvedWebcamVideoUrl : null}
trimRegions={trimRegions}
Expand Down Expand Up @@ -6338,6 +6352,10 @@ export default function VideoEditor() {
cropRegion={cropRegion}
onCropChange={setCropRegion}
aspectRatio={aspectRatio}
cursorFollow={cursorFollowCrop}
onCursorFollowChange={setCursorFollowCrop}
Comment on lines +6355 to +6356
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Jun 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Cancel should roll back cursor-follow edits too.

These props let the crop modal mutate cursorFollowCrop immediately, but handleCancelCropEditor() only restores cropRegion. If the user toggles Track cursor or changes safe zone/smoothness and then cancels, those edits still stick.

💡 Minimal fix 
+	const cursorFollowSnapshotRef = useRef<CursorFollowCropSettings | null>(null);
+
 	const handleOpenCropEditor = useCallback(() => {
 		cropSnapshotRef.current = { ...cropRegion };
+		cursorFollowSnapshotRef.current = { ...cursorFollowCrop };
 		setShowCropModal(true);
-	}, [cropRegion]);
+	}, [cropRegion, cursorFollowCrop]);

 	const handleCancelCropEditor = useCallback(() => {
 		if (cropSnapshotRef.current) {
 			setCropRegion(cropSnapshotRef.current);
 		}
+		if (cursorFollowSnapshotRef.current) {
+			setCursorFollowCrop(cursorFollowSnapshotRef.current);
+		}
 		setShowCropModal(false);
-	}, []);
+	}, []);
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/components/video-editor/VideoEditor.tsx` around lines 6355 - 6356, The
cancel path only restores cropRegion but not the mutable cursor-follow state, so
when the crop modal is opened you should snapshot the current cursorFollowCrop
(e.g., prevCursorFollowCrop) and any related fields (safe zone, smoothness) and
then in handleCancelCropEditor() restore cursorFollowCrop via
setCursorFollowCrop(prevCursorFollowCrop) (and clear the snapshot after). Add
the snapshot creation where the editor opens and use the snapshot in
handleCancelCropEditor to roll back cursorFollowCrop changes.

cursorTelemetry={cursorTelemetry}
currentTimeMs={currentTime * 1000}
Comment on lines +6357 to +6358
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Jun 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Use the same telemetry variant as preview/export.

The crop modal gets raw cursorTelemetry, while VideoPlayback and both export paths use effectiveCursorTelemetry after normalization/loop handling. With looped cursor playback or timeline-adjusted telemetry, the crop editor can preview a different camera path than the actual render.

💡 Minimal fix 
-							cursorTelemetry={cursorTelemetry}
+							cursorTelemetry={effectiveCursorTelemetry}
 							currentTimeMs={currentTime * 1000}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
cursorTelemetry={cursorTelemetry}
currentTimeMs={currentTime * 1000}
cursorTelemetry={effectiveCursorTelemetry}
currentTimeMs={currentTime * 1000}
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/components/video-editor/VideoEditor.tsx` around lines 6357 - 6358, The
crop modal is being passed raw cursorTelemetry which can differ from the
normalized/loop-aware path used by VideoPlayback and export; change the prop
passed to the crop editor from cursorTelemetry to the already-computed
effectiveCursorTelemetry (the same value used by VideoPlayback/export) so the
crop preview uses the identical telemetry variant; locate the prop usage in
VideoEditor (prop name cursorTelemetry/currentTimeMs) and replace it to supply
effectiveCursorTelemetry (or call the helper that computes it) so preview/export
paths match.

/>
<div className="mt-6 flex justify-end">
<Button
Expand Down
63 changes: 63 additions & 0 deletions src/components/video-editor/VideoPlayback.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,12 @@ import {
resetCursorFollowCamera,
SNAP_TO_EDGES_RATIO_AUTO,
} from "./videoPlayback/cursorFollowCamera";
import {
type CursorFollowCropState,
computeCursorFollowCrop,
createCursorFollowCropState,
resetCursorFollowCropState,
} from "./videoPlayback/cursorFollowCrop";
import { clampFocusToStage as clampFocusToStageUtil } from "./videoPlayback/focusUtils";
import { layoutVideoContent as layoutVideoContentUtil } from "./videoPlayback/layoutUtils";
import { updateOverlayIndicator } from "./videoPlayback/overlayUtils";
Expand Down Expand Up @@ -350,6 +356,7 @@ interface VideoPlaybackProps {
padding?: Padding | number;
frame?: string | null;
cropRegion?: import("./types").CropRegion;
cursorFollowCrop?: import("./types").CursorFollowCropSettings;
webcam?: WebcamOverlaySettings;
webcamVideoPath?: string | null;
trimRegions?: TrimRegion[];
Expand Down Expand Up @@ -433,6 +440,7 @@ const VideoPlayback = forwardRef<VideoPlaybackRef, VideoPlaybackProps>(
padding = DEFAULT_PADDING,
frame = null,
cropRegion,
cursorFollowCrop,
webcam,
webcamVideoPath,
trimRegions = [],
Expand Down Expand Up @@ -611,6 +619,12 @@ const VideoPlayback = forwardRef<VideoPlaybackRef, VideoPlaybackProps>(
const cursorFollowCameraRef = useRef<CursorFollowCameraState>(
createCursorFollowCameraState(),
);
const cursorFollowCropRef = useRef(cursorFollowCrop);
const cursorFollowCropStateRef = useRef<CursorFollowCropState>(
createCursorFollowCropState(),
);
const baseCropRegionRef = useRef(cropRegion);
const effectiveCropRegionRef = useRef(cropRegion);

const initializePixiRenderer = useCallback(
async (
Expand Down Expand Up @@ -1534,6 +1548,17 @@ const VideoPlayback = forwardRef<VideoPlaybackRef, VideoPlaybackProps>(
connectedZoomEasingRef.current = connectedZoomEasing;
}, [connectedZoomEasing]);

useEffect(() => {
cursorFollowCropRef.current = cursorFollowCrop;
resetCursorFollowCropState(cursorFollowCropStateRef.current);
}, [cursorFollowCrop?.enabled, cursorFollowCrop?.safeZoneRatio, cursorFollowCrop?.smoothness, cursorFollowCrop?.trackTextCursor]);

useEffect(() => {
baseCropRegionRef.current = cropRegion ?? { x: 0, y: 0, width: 1, height: 1 };
effectiveCropRegionRef.current = baseCropRegionRef.current;
resetCursorFollowCropState(cursorFollowCropStateRef.current);
}, [cropRegion]);

useEffect(() => {
cursorTelemetryRef.current = cursorTelemetry;
// Push to extension host for query APIs
Expand Down Expand Up @@ -2194,6 +2219,44 @@ const VideoPlayback = forwardRef<VideoPlaybackRef, VideoPlaybackProps>(
return;
}

// Cursor-follow crop: per-frame viewport pan within the source video.
// When enabled, recompute the effective crop top-left from cursor
// telemetry and shift the video sprite to keep the cursor framed.
const followSettings = cursorFollowCropRef.current;
const baseCrop = baseCropRegionRef.current;
const sprite = videoSpriteRef.current;
const lockedDims = lockedVideoDimensionsRef.current;
if (followSettings?.enabled && sprite && lockedDims && baseCrop) {
const effectiveCrop = computeCursorFollowCrop(
cursorFollowCropStateRef.current,
cursorTelemetryRef.current,
currentTimeRef.current,
baseCrop,
followSettings,
);
effectiveCropRegionRef.current = effectiveCrop;
const fullVideoDisplayWidth = lockedDims.width * baseScaleRef.current;
const fullVideoDisplayHeight = lockedDims.height * baseScaleRef.current;
const dx = (effectiveCrop.x - baseCrop.x) * fullVideoDisplayWidth;
const dy = (effectiveCrop.y - baseCrop.y) * fullVideoDisplayHeight;
sprite.position.set(
baseOffsetRef.current.x - dx,
baseOffsetRef.current.y - dy,
);
cropBoundsRef.current = {
startX: effectiveCrop.x * lockedDims.width,
endX: effectiveCrop.x * lockedDims.width + effectiveCrop.width * lockedDims.width,
startY: effectiveCrop.y * lockedDims.height,
endY: effectiveCrop.y * lockedDims.height + effectiveCrop.height * lockedDims.height,
};
if (baseMaskRef.current.sourceCrop) {
baseMaskRef.current.sourceCrop = { ...effectiveCrop };
}
Comment on lines +2135 to +2137
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Jun 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

sourceCrop may never be initialized when cursor-follow is enabled.

The code only mutates baseMaskRef.current.sourceCrop if it already exists, but nothing initializes it. If cursor-follow is enabled from the start, downstream rendering logic expecting sourceCrop won't receive the effective crop coordinates.

Consider initializing sourceCrop unconditionally:

🐛 Proposed fix 
-				if (baseMaskRef.current.sourceCrop) {
-					baseMaskRef.current.sourceCrop = { ...effectiveCrop };
-				}
+				baseMaskRef.current.sourceCrop = { ...effectiveCrop };
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if (baseMaskRef.current.sourceCrop) {
baseMaskRef.current.sourceCrop = { ...effectiveCrop };
}
baseMaskRef.current.sourceCrop = { ...effectiveCrop };
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/components/video-editor/VideoPlayback.tsx` around lines 2135 - 2137, The
code currently only assigns to baseMaskRef.current.sourceCrop when it already
exists, so when cursor-follow starts enabled that field may be undefined and
downstream rendering misses the effective crop; update the logic around
baseMaskRef (referencing baseMaskRef.current, sourceCrop, and effectiveCrop) to
always initialize sourceCrop to the effectiveCrop (e.g., set
baseMaskRef.current.sourceCrop = { ...effectiveCrop } unconditionally or ensure
initialization before cursor-follow usage), ensuring the property exists
whenever cursor-follow or rendering logic reads it; keep the assignment in the
same function where effectiveCrop is computed so you don't change call sites.

} else if (sprite) {
effectiveCropRegionRef.current = baseCrop;
sprite.position.set(baseOffsetRef.current.x, baseOffsetRef.current.y);
}

const { region, strength, blendedScale, transition } = findDominantRegion(
zoomRegionsRef.current,
currentTimeRef.current,
Expand Down
26 changes: 26 additions & 0 deletions src/components/video-editor/projectPersistence.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ import {
type ClipRegion,
type CropRegion,
type CursorClickEffectStyle,
type CursorFollowCropPreviewMode,
type CursorFollowCropSettings,
type CursorStyle,
DEFAULT_ANNOTATION_POSITION,
DEFAULT_ANNOTATION_SIZE,
Expand All @@ -45,6 +47,7 @@ import {
DEFAULT_CURSOR_CLICK_EFFECT_DURATION_MS,
DEFAULT_CURSOR_CLICK_EFFECT_OPACITY,
DEFAULT_CURSOR_CLICK_EFFECT_SCALE,
DEFAULT_CURSOR_FOLLOW_CROP,
DEFAULT_CURSOR_STYLE,
DEFAULT_CURSOR_SWAY,
DEFAULT_FIGURE_DATA,
Expand Down Expand Up @@ -129,6 +132,7 @@ export interface ProjectEditorState {
/** Selected frame ID (e.g. "recordly.frames/browser-dark"), or null for none */
frame: string | null;
cropRegion: CropRegion;
cursorFollowCrop: CursorFollowCropSettings;
zoomRegions: ZoomRegion[];
trimRegions: TrimRegion[];
clipRegions: ClipRegion[];
Expand Down Expand Up @@ -816,6 +820,27 @@ export function normalizeProjectEditor(editor: Partial<ProjectEditorState>): Pro
const cropWidth = clamp(rawCropWidth, 0.01, 1 - cropX);
const cropHeight = clamp(rawCropHeight, 0.01, 1 - cropY);

const rawCursorFollowCrop = (editor as Partial<ProjectEditorState>).cursorFollowCrop;
const cursorFollowCropPreviewMode: CursorFollowCropPreviewMode =
rawCursorFollowCrop?.previewMode === "output" ? "output" : "source";
const normalizedCursorFollowCrop: CursorFollowCropSettings = {
enabled:
typeof rawCursorFollowCrop?.enabled === "boolean"
? rawCursorFollowCrop.enabled
: DEFAULT_CURSOR_FOLLOW_CROP.enabled,
safeZoneRatio: isFiniteNumber(rawCursorFollowCrop?.safeZoneRatio)
? clamp(rawCursorFollowCrop.safeZoneRatio, 0, 0.49)
: DEFAULT_CURSOR_FOLLOW_CROP.safeZoneRatio,
smoothness: isFiniteNumber(rawCursorFollowCrop?.smoothness)
? clamp(rawCursorFollowCrop.smoothness, 0, 1)
: DEFAULT_CURSOR_FOLLOW_CROP.smoothness,
previewMode: cursorFollowCropPreviewMode,
trackTextCursor:
typeof rawCursorFollowCrop?.trackTextCursor === "boolean"
? rawCursorFollowCrop.trackTextCursor
: DEFAULT_CURSOR_FOLLOW_CROP.trackTextCursor,
};

const webcam: Partial<WebcamOverlaySettings> =
editor.webcam && typeof editor.webcam === "object" ? editor.webcam : {};
const webcamSourcePath = typeof webcam.sourcePath === "string" ? webcam.sourcePath : null;
Expand Down Expand Up @@ -977,6 +1002,7 @@ export function normalizeProjectEditor(editor: Partial<ProjectEditorState>): Pro
width: cropWidth,
height: cropHeight,
},
cursorFollowCrop: normalizedCursorFollowCrop,
zoomRegions: normalizedZoomRegions,
trimRegions: normalizedTrimRegions,
clipRegions: normalizedClipRegions,
Expand Down
26 changes: 26 additions & 0 deletions src/components/video-editor/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -497,6 +497,32 @@ export const DEFAULT_CROP_REGION: CropRegion = {
height: 1,
};

export type CursorFollowCropPreviewMode = "source" | "output";

export interface CursorFollowCropSettings {
enabled: boolean;
/** Safe-zone inset 0..0.49 — fraction of viewport edge before camera pans. */
safeZoneRatio: number;
/** 0..1, smoothing applied per frame (higher = slower follow, less jitter). */
smoothness: number;
/** Editor-only: which view to show in the crop panel. */
previewMode: CursorFollowCropPreviewMode;
/**
* When true: viewport locks to the text I-beam position while typing and
* only aggressively follows the mouse when it's actively moving. Mouse
* always wins; transitions are debounced to avoid jarring cuts.
*/
trackTextCursor: boolean;
}

export const DEFAULT_CURSOR_FOLLOW_CROP: CursorFollowCropSettings = {
enabled: false,
safeZoneRatio: 0.25,
smoothness: 0.5,
previewMode: "source",
trackTextCursor: false,
};

export interface Padding {
top: number;
bottom: number;
Expand Down
Loading