Skip to content

fix: allow negative POINT coordinates so off-screen elements still point - #129

Open
TUARAN wants to merge 1 commit into
farzaa:mainfrom
TUARAN:fix/allow-negative-pointing-coordinates
Open

fix: allow negative POINT coordinates so off-screen elements still point#129
TUARAN wants to merge 1 commit into
farzaa:mainfrom
TUARAN:fix/allow-negative-pointing-coordinates

Conversation

@TUARAN

@TUARAN TUARAN commented Aug 4, 2026

Copy link
Copy Markdown

Summary

parsePointingCoordinates now accepts an optional minus sign on the x/y values in [POINT:x,y:label:screenN] tags, so slightly negative coordinates from Claude no longer make the parser reject the tag and skip pointing.

Issue

Fixes #128 — "Cursor pointing is skipped when Claude returns a slightly negative coordinate".

Root cause

The pattern in leanring-buddy/CompanionManager.swift used \d+ for both coordinates. Claude occasionally returns slightly negative coordinates (e.g. [POINT:-1,42:button]) for elements near the screen edge — a case ElementLocationDetector already documents. The \d+ groups cannot match a leading -, so the whole tag fails to parse, the raw tag is left in the spoken text, and pointing is skipped even though the downstream code already clamps out-of-bounds coordinates to the screenshot bounds.

Verification

Behavior verified with the same regex engine the app uses (NSRegularExpression / ICU) via a standalone harness, before vs after the one-line change:

Input Old pattern New pattern
[POINT:1100,42:color inspector] matched (1100,42) matched (1100,42)
[POINT:-1,42:button] no match matched (-1,42)
[POINT:10,-5:button:screen2] no match matched (10,-5) screen=2
[POINT:-3,-7:menu] no match matched (-3,-7)
[POINT:none] matched, no coordinate matched, no coordinate
no tag no match no match

swiftc -parse leanring-buddy/CompanionManager.swift exits 0 after the change.

Before/after behavior

  • Before: [POINT:-1,42:button]parsePointingCoordinates returns coordinate: nil, the raw tag stays in the spoken text, and no cursor flight is scheduled.
  • After: the same tag → coordinate: (-1, 42), tag removed from spoken text, and the existing clamp (max(0, min(pointCoordinate.x, screenshotWidth)) in CompanionManager.swift) maps the negative value to the screenshot edge.

Risk and scope

One-line regex change. No behavior change for non-negative coordinates, [POINT:none], or responses without a tag. No new dependencies.

Not tested

Full app run (requires a real macOS UI with screen-recording/audio permissions; xcodebuild is intentionally avoided per the repo instructions).

Claude occasionally returns slightly negative coordinates for elements near the screen edge, and ElementLocationDetector already documents that out-of-range coordinates occur. The parser's \d+ groups rejected them, so pointing was skipped and the raw [POINT:...] tag leaked into the spoken text. Accept an optional minus sign so the existing clamp path handles the bounds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cursor pointing is skipped when Claude returns a slightly negative coordinate

1 participant