ThisThat

Point at anything. Let Codex understand it.

ThisThat is a Codex-native macOS visual context layer. It combines a signed menu-bar companion app with a local MCP server so Codex can understand what you mean when you say "this", "that", "the highlighted area", or "the thing on screen".

ThisThat is built from the best part of TipTour: the pointer/highlighter idea inspired by Google DeepMind's Gemini-powered AI pointer work. TipTour was built around Gemini Live and required a Gemini API key. ThisThat removes the Gemini dependency and turns the highlighter into a local visual context bridge for Codex.

It does not try to replace Codex's existing tools. It complements them:

• ThisThat captures and explains visual context.
• OpenAI Chrome skill handles browser interaction.
• OpenAI Computer Use handles native desktop interaction.
• Codex/ChatGPT remains the reasoning and planning layer.

This project is not affiliated with Google, Gemini, DeepMind, OpenAI, or Codex. It is an independent open-source fork/adaptation of TipTour for Codex workflows.

Why This Exists

Codex is powerful, but desktop work often starts with an imprecise human reference:

• "Use this button."
• "Fix that part of the image."
• "What is happening in this game?"
• "Change the highlighted text."
• "Click the control inside this CAD canvas."
• "Compare these two areas."

Screenshots help, but screenshots are blunt. They lack task context, screen coordinates, app/window metadata, source URLs, file paths, highlight history, OCR, accessibility labels, visual crops, and a reliable way to connect the user's next instruction to the exact region they meant.

ThisThat turns pointing into structured context.

What It Does

Hold Control+Shift and drag over anything on your Mac. ThisThat records:

• the highlighted region
• the active app and window
• screen and window coordinates
• crop images and full-frame references
• best-effort source locators: browser page URLs, inferred asset/link URLs, Finder selections, and native document paths when macOS exposes them
• OCR and local visual target data
• accessibility labels when available
• screenshot hashes and visual-change metadata
• compact context files optimized for Codex ingestion

Codex can then use ThisThat MCP tools to inspect the highlighted region, ask for a focused crop, search nearby visual targets, read text, tile the image, probe pixels, compare before/after frames, use exact source paths/URLs when available, or wait for a visual change after Chrome/Computer Use performs an action.

Architecture

flowchart LR
  User["User paints Control+Shift highlight"] --> App["ThisThat.app signed menu-bar companion"]
  App --> Capture["ScreenCaptureKit + AX + OCR + local detector"]
  Capture --> Bridge["Token-protected localhost bridge"]
  Bridge --> MCP["thisthat MCP stdio server"]
  MCP --> Codex["Codex"]
  Codex --> Chrome["OpenAI Chrome skill"]
  Codex --> ComputerUse["OpenAI Computer Use"]

Loading

The split is intentional:

• The macOS app owns Screen Recording, Screen Content, Accessibility, overlay UI, highlighter state, visual capture, and local diagnostics.
• The MCP daemon is launched by Codex and talks to the app over localhost with a per-run token.
• Codex consumes the visual context and chooses the right action surface.

This avoids continuous screenshot streaming and avoids making a Node MCP process responsible for macOS privacy permissions.

TipTour Lineage

ThisThat should be understood as a Codex-focused continuation of TipTour, not as an unrelated rewrite.

TipTour explored a Gemini-style pointer assistant: a macOS companion that could use screen context and a pointer/highlighter interaction to help users through apps. That direction tracks the broader AI pointer idea described by Google DeepMind: the pointer should carry enough visual and semantic context for the model to understand what the user is referring to.

ThisThat keeps that interaction model but changes the AI route:

Area	TipTour	ThisThat
AI route	Gemini / Gemini Live	Codex / ChatGPT
API key	Gemini API key required	No Gemini API key
Primary role	Assistant that can guide/control	Visual context layer for Codex
Browser work	App-specific integrations	OpenAI Chrome skill
Desktop action	Native action stack	OpenAI Computer Use
Best retained idea	Pointer/highlighter context	Pointer/highlighter context optimized for Codex

Use Cases

ThisThat is useful when the target is hard to describe in text:

• image editing: "remove this artifact", "match this color", "crop around this object"
• CAD and Fusion 360: "select this edge", "inspect this tiny control", "what is this panel showing?"
• games: "what should I do here?", "track this UI element", "compare the before/after state"
• browser work: "use this part of the page", then let the Chrome skill act
• native macOS apps: "change this field", then let Computer Use act
• documents and screenshots: "rewrite this highlighted paragraph", "summarize this section"
• debugging visual UIs: "verify this button moved", "wait until this spinner disappears"
• accessibility-poor apps: inspect unlabeled controls with OCR, crops, grids, and pixel probes

MCP Tool Surface

The public surface is visual-first:

• launch_companion
• bridge_health
• bridge_diagnostics
• codex_readiness
• permission_preflight
• observe_desktop
• get_highlight_context
• highlight_visual_frame
• list_windows
• visual_grounding_frame
• annotated_visual_frame
• window_visual_frame
• visual_inspection_bundle
• visual_scene_scan
• visual_timeline_scan
• crop_visual_observation
• tile_visual_observation
• read_visual_text
• search_visual_targets
• target_visual_frame
• coordinate_grid_frame
• probe_visual_observation
• find_similar_visual_regions
• visual_workflow_status
• pointer_status
• list_visual_observations
• wait_for_visual_change
• diff_visual_observations
• cleanup_observations
• open_visual_test_harness
• run_torture_test

ThisThat intentionally does not expose broad desktop-control tools such as generic click/type/drag, browser automation, clipboard mutation, or task approval workflows. Codex should use ThisThat to understand the visual target, then use Chrome or Computer Use to act.

Privacy Model

ThisThat is local-first.

• It does not stream continuous screenshots.
• It captures visual context when Codex calls a visual tool or when you create a highlight.
• The bridge token is generated per app run and stored locally under ~/Library/Application Support/ThisThat/CodexBridge/bridge-token.
• Diagnostic logs are local JSONL files with redaction.
• No Gemini API key is used.

Codex may still receive screenshots/crops when a ThisThat MCP tool returns image context. Treat highlighted regions as potentially sensitive.

Build Requirements

• macOS 14+
• Xcode 16+
• Node 20+
• A local Apple Development or Developer ID signing identity for reliable macOS privacy permissions

Build And Run

./script/build_and_run.sh --verify

If macOS permissions are not ready:

./script/open_permission_setup.sh --wait 300 --interval 2 --restart-before-request --restart-after-ready

Install the MCP entry pinned to the built app:

./scripts/install-thisthat-mcp.sh --force --app-path "$PWD/build/DerivedData/Build/Products/Debug/ThisThat.app"

Restart Codex after installation so it reloads MCP configuration.

Verification

npm --prefix mcp test
./script/verify_all.sh --no-live

For live visual validation:

THISTHAT_RUN_LIVE=1 ./script/verify_all.sh

The live harness uses deterministic fake visual surfaces so Codex can stress test visual tools without touching real user apps or files.

Documentation

• MCP architecture and tool guide
• GitHub Pages landing page source

License And Attribution

MIT. ThisThat is based on TipTour and preserves public attribution to TipTour and its upstream lineage in the license and documentation.