Skip to content

feat(agent): finish agent mode phases 2-6 - #36

Merged
aryanjsx merged 13 commits into
mainfrom
feat/agent-mode-phases-2-6
Nov 8, 2026
Merged

aryanjsx merged 13 commits into
mainfrom
feat/agent-mode-phases-2-6

Conversation

@aryanjsx

@aryanjsx aryanjsx commented Nov 8, 2026

Copy link
Copy Markdown
Owner

What does this PR do?

Finishes docs/design/2026-11-08-agent-mode-plan.md, so Agent mode can do tasks on the computer from text or voice. Each part is its own commit.

Phase 2: attachments, images and web search reach the agent

  • In Agent mode, these no longer fall back to plain chat.
  • Documents are folded into each message, using the same budget as direct chat.
  • The latest message's images reach the agent through AgentContext.metadata["input_images"].
  • Web results join the agent's system prompt, so answers keep their numbered sources.

Phase 3 leftovers

  • Plan checklist: the agent calls a new task_plan tool before multi-step work and as steps finish. The chat shows the latest plan as a checklist.
  • Undo: Agent mode's file_write saves the old contents under ~/.locus/undo, and the tool card shows Undo. Undo only applies while the file still holds what the agent wrote, so later edits are never lost.
  • Always allow: a lasting rule for change-level steps, scoped to one tool or tool:action. Rules are kept in ~/.locus/agent_permissions.json, and Settings > Agent mode lists and removes them. Commands and code always ask.
  • Previews: approvals can carry a preview from the tool, such as a file diff.
  • Continue: a stopped task says what was finished and shows Continue, which picks the task up again.

Phase 4: desktop control tools (seven tools in locus.tools.desktop)

  • Tools: open_path, app_control, window_control, system_control, clipboard, notify and screenshot.
  • Platforms: Windows is fully supported. macOS and Linux cover what their standard tools allow.
  • Wi-Fi and Bluetooth: switching on Windows needs no admin rights.
  • Reminders: saved, restored when locus serve starts, and reminders missed while closed show right away.
  • No command injection: commands run without a shell, and model input reaches PowerShell only as LOCUS_ARG_* environment variables.
  • Per-call risk: tools can define risk_for(arguments), so listing windows doesn't ask but closing an app does.

Phase 5: hands-free voice (Settings > Speech)

  • Shortcut: Ctrl+Shift+L (Cmd+Shift+L on a Mac) works in the window, and system-wide in the desktop app, which raises its window first.
  • Listening: a silence detector sends the message when you stop talking.
  • Replies: read aloud, even when "Speak replies automatically" is off.
  • Interrupting: pressing the shortcut again stops a spoken reply.

Phase 6: developer mode

  • Project folder: a Project button beside Agent picks a folder, and requests use mode: "developer" with project_dir.
  • Scope: file, git and shell tools stay inside the project. Edits show a diff before they're written and can be undone.
  • Commands: tests, linters and git status/diff/log/show run without asking. Any other command, or any command with shell operators, asks every time. Commits always ask.
  • Instructions: the agent is told to work on a branch, run the tests after changes, fix failures up to 3 times, and never push.

Security inventory: the new tools are added to DEFAULT_TOOL_CAPABILITIES, so they don't fail closed as system:admin.

Things a reviewer should know

  • Capability policy: it's off by default. With default_deny on, the assistant and developer agents get only the read baseline, so writing, commands and desktop control need explicit grants.
  • Windows permissions: the desktop app gains core:window:allow-show, allow-unminimize and allow-set-focus for the hands-free shortcut.
  • Not built:
    • a wake word
    • voice answers to approvals
    • moving and snapping windows
    • window control on Wayland
    • sending screenshots to vision models
    • apply_patch in developer mode (it has no folder limit, so edits use file_write with a diff instead)

How was this tested?

  • New tests:
    • tests/tools/test_desktop_tools.py, test_undo_journal.py and test_task_plan.py
    • tests/server/test_developer_mode.py
    • additions to test_agent_mode.py and test_agent_approvals.py
    • frontend/src/lib/vad.test.ts, agent-stop.test.ts and TaskPlan.test.ts
    • more cases in ToolCallCard.test.ts
  • Frontend: 144 vitest tests pass, and tsc -b is clean.
  • On a Windows 11 PC, with only non-disruptive actions:
    • listing apps and windows
    • reading the volume, then setting it to the same value
    • reading the Wi-Fi and Bluetooth state
    • showing a notification
    • rejecting bad input
  • Real orchestrator with a scripted model:
    • desktop tools: reading the volume didn't ask, and opening Notepad asked and was denied
    • developer mode in a temporary git repo: git status and git diff didn't ask, a file edit asked with a diff and was applied, and rm asked and was denied
  • UI: in the browser GUI with no backend, the new Agent and Project buttons and the Settings sections render.
  • Full Python suites: run per folder: server, tools, agents, cli, core, speech and security. Compared file by file with main, every failure on this branch also fails on main, mostly because the locus_rust extension isn't installed locally. The one exception is tests/tools/test_tool_timeout.py::test_repeated_timeouts_use_bounded_workers, which failed in the full run but passes alone, so it depends on test order.
  • Hang on main: running tests/tools before tests/agents hangs at test_agent_routing_matrix.py::test_model_passthrough[llama3:70b-native_openhands]. That happens on main too.
  • Not tried: the desktop app end to end with a real model, and hands-free voice with a real microphone. Ollama and a speech backend weren't available.

Checklist

  • Tests pass (uv run pytest tests/ -v): new and changed suites pass; the remaining failures also fail on main
  • Linter passes (ruff check src/ and the new tests)
  • Formatter passes on changed files
  • New/changed public API has docstrings
  • Follows registry pattern (if adding new component)
  • Documentation updated (if applicable)

Agent mode could work with files and commands but not the computer
itself. Add seven tools in locus.tools.desktop, Windows first:

- open_path: open a file, folder or http(s) link in its default app.
- app_control: list apps with windows, open an app by name, or ask it
  to close like clicking X.
- window_control: list, focus, minimize, maximize and restore windows
  by part of the title.
- system_control: read and set the volume, mute, laptop brightness,
  dark and light mode, and lock the screen.
- clipboard: read and write text.
- notify: a desktop notification now, or a reminder up to 24 hours
  later while Locus runs.
- screenshot: the whole screen as a PNG in ~/.locus/screenshots.

Commands run without a shell, and values from the model reach
PowerShell only through LOCUS_ARG_* environment variables, so a title
or name can't inject a command. No new dependencies.

Tools can now define risk_for(arguments), so approvals depend on the
call: listing windows or reading the volume doesn't ask, while closing
an app, opening a program or reading the clipboard does. The approval
card describes these steps in words.
Agent mode answered any conversation with attachments or web search as
plain chat, because agents only took the latest message's text.

- The agent gets every message with attached documents folded in, using
  the same budget as direct chat.
- The latest message's images go to the agent through
  AgentContext.metadata["input_images"], which BaseAgent puts on the
  final user message for vision models.
- Web search stays a pre-search, so answers keep their numbered sources;
  the results join the agent's system prompt.
- In Agent mode these requests now reach the agent. Other requests route
  as before.
Add a task_plan tool the Agent mode agent calls before a multi-step task
and again as steps finish. The chat shows the latest plan above the
answer as a checklist (pending, in progress, done, skipped) instead of a
tool card, and the status line reads "Planning..." while it updates.
Planning never asks for approval.
Before Agent mode writes a file, it now saves the old contents (or notes
that the file is new) in an undo journal under ~/.locus/undo. The tool
card shows an Undo button, and POST /v1/chat/undo/{id} puts the old
version back or deletes a file the agent created.

Undo only applies while the file still holds what the agent wrote, so
edits made afterwards are never lost. Files over 10 MB aren't backed up,
and the journal keeps the last 100 changes.
Approvals could cover one call or one task, but every new task asked
again for the same everyday steps.

- The approval card offers Always allow for change-level steps. The
  rule covers that tool, or one action of it (system_control:set_volume
  doesn't also allow lock_screen), and is saved in
  ~/.locus/agent_permissions.json.
- Commands and code (run-level steps) never get lasting rules.
- Settings has an Agent mode section listing the rules, each with
  Remove. GET /v1/chat/permissions and DELETE /v1/chat/permissions/{key}
  back it.
- Approval requests can carry a preview from the tool's preview_for(),
  shown above the buttons.
- Server tests keep the rules file in a temp folder.
Stopping an Agent mode task left "(Generation stopped)", so asking it to
go on meant starting over. Now the stopped reply says what was finished
("Done so far: Write a.txt; Mute the speakers.") and shows Continue.
Continue turns Agent mode on and sends a follow-up asking the agent to
pick up where it stopped without redoing finished steps. The summary
stays in the conversation, so the agent knows what's done.
system_control gains get_radios (reads whether Wi-Fi and Bluetooth are
on, without asking) and wifi_on, wifi_off, bluetooth_on and
bluetooth_off (which ask first).

- Windows uses Windows.Devices.Radios through PowerShell, so it doesn't
  need admin rights.
- macOS uses networksetup for Wi-Fi and blueutil, when installed, for
  Bluetooth.
- Linux uses nmcli and rfkill.
window_control worked only on Windows. It now shares one flow (list
windows, match part of the title, act) with a backend per OS:

- macOS uses System Events through osascript; the error explains how to
  allow Locus under Accessibility.
- Linux uses wmctrl on X11 and says to install it when it's missing.
Reminders lived only in the running app, so closing Locus lost them.
They're now saved in ~/.locus/reminders.json. When locus serve starts,
it schedules them again and shows any that came due while Locus was
closed, marked as missed.

notify also gains list_reminders and cancel_reminder, and reminders can
be up to 30 days out (was 24 hours).
Voice was push-to-talk only: click the mic, talk, click again, then
send. With Hands-free voice on (Settings > Speech):

- Ctrl+Shift+L (Cmd+Shift+L on a Mac) starts listening. It works in the
  window, and from any app in the desktop app, which raises Locus first.
- A silence detector learns the room's noise for the first moments and
  stops recording when the speaker has been quiet for 1.2 s. If nobody
  speaks for 8 s it cancels; turns are capped at 30 s.
- The transcript is sent like a typed message, and the reply is read
  aloud even when "Speak replies automatically" is off.
- Pressing the shortcut again while a reply is being spoken interrupts
  it and listens; pressing it while listening sends straight away.

The desktop app can now show, unminimize and focus its window so the
global shortcut can bring it forward.
A Project button next to Agent picks a folder (a native folder picker in
the desktop app). Chat then runs a developer agent kept inside that
project:

- file_read and file_write (with Undo) are limited to the project, and
  every file change shows a diff in the approval before it's written.
- shell_exec runs in the project. Tests, linters and git status, diff,
  log and show run without asking; anything else, and anything with
  shell operators (; | > $( ) and similar), asks every time.
- git tools default to the project and refuse paths outside it.
  Commits always ask.
- The agent is told to plan, read before editing, work on a new branch
  instead of main, run the tests after changes and fix failures up to 3
  times, and never push.

The request field is mode "developer" with project_dir; a drive root,
the home folder or a missing folder gets a 400 saying why. One agent is
kept per project.

Tools can now define risk_for before the name lists apply, so developer
mode's shell can let known checks through. Anything marked
requires_confirmation still always asks.
Built-in tools missing from DEFAULT_TOOL_CAPABILITIES are treated as
system:admin and fail closed, and test_every_registered_builtin_has_an_
explicit_inventory_entry caught that the new tools weren't listed.

task_plan is reviewed-safe (no capabilities, like think). open_path and
app_control can start programs (code:execute), system_control changes
computer settings (system:admin), clipboard and screenshot read and
write data (file:read, file:write), notify schedules reminders
(schedule:create), and window_control is tool:invoke.
@aryanjsx
aryanjsx merged commit 4866e3c into main Nov 8, 2026
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant