Repository navigation
feat(agent): finish agent mode phases 2-6 - #36
Merged
Merged
Conversation
Agent mode could work with files and commands but not the computer itself. Add seven tools in locus.tools.desktop, Windows first: - open_path: open a file, folder or http(s) link in its default app. - app_control: list apps with windows, open an app by name, or ask it to close like clicking X. - window_control: list, focus, minimize, maximize and restore windows by part of the title. - system_control: read and set the volume, mute, laptop brightness, dark and light mode, and lock the screen. - clipboard: read and write text. - notify: a desktop notification now, or a reminder up to 24 hours later while Locus runs. - screenshot: the whole screen as a PNG in ~/.locus/screenshots. Commands run without a shell, and values from the model reach PowerShell only through LOCUS_ARG_* environment variables, so a title or name can't inject a command. No new dependencies. Tools can now define risk_for(arguments), so approvals depend on the call: listing windows or reading the volume doesn't ask, while closing an app, opening a program or reading the clipboard does. The approval card describes these steps in words.
Agent mode answered any conversation with attachments or web search as plain chat, because agents only took the latest message's text. - The agent gets every message with attached documents folded in, using the same budget as direct chat. - The latest message's images go to the agent through AgentContext.metadata["input_images"], which BaseAgent puts on the final user message for vision models. - Web search stays a pre-search, so answers keep their numbered sources; the results join the agent's system prompt. - In Agent mode these requests now reach the agent. Other requests route as before.
Add a task_plan tool the Agent mode agent calls before a multi-step task and again as steps finish. The chat shows the latest plan above the answer as a checklist (pending, in progress, done, skipped) instead of a tool card, and the status line reads "Planning..." while it updates. Planning never asks for approval.
Before Agent mode writes a file, it now saves the old contents (or notes
that the file is new) in an undo journal under ~/.locus/undo. The tool
card shows an Undo button, and POST /v1/chat/undo/{id} puts the old
version back or deletes a file the agent created.
Undo only applies while the file still holds what the agent wrote, so
edits made afterwards are never lost. Files over 10 MB aren't backed up,
and the journal keeps the last 100 changes.
Approvals could cover one call or one task, but every new task asked
again for the same everyday steps.
- The approval card offers Always allow for change-level steps. The
rule covers that tool, or one action of it (system_control:set_volume
doesn't also allow lock_screen), and is saved in
~/.locus/agent_permissions.json.
- Commands and code (run-level steps) never get lasting rules.
- Settings has an Agent mode section listing the rules, each with
Remove. GET /v1/chat/permissions and DELETE /v1/chat/permissions/{key}
back it.
- Approval requests can carry a preview from the tool's preview_for(),
shown above the buttons.
- Server tests keep the rules file in a temp folder.
Stopping an Agent mode task left "(Generation stopped)", so asking it to
go on meant starting over. Now the stopped reply says what was finished
("Done so far: Write a.txt; Mute the speakers.") and shows Continue.
Continue turns Agent mode on and sends a follow-up asking the agent to
pick up where it stopped without redoing finished steps. The summary
stays in the conversation, so the agent knows what's done.
system_control gains get_radios (reads whether Wi-Fi and Bluetooth are on, without asking) and wifi_on, wifi_off, bluetooth_on and bluetooth_off (which ask first). - Windows uses Windows.Devices.Radios through PowerShell, so it doesn't need admin rights. - macOS uses networksetup for Wi-Fi and blueutil, when installed, for Bluetooth. - Linux uses nmcli and rfkill.
window_control worked only on Windows. It now shares one flow (list windows, match part of the title, act) with a backend per OS: - macOS uses System Events through osascript; the error explains how to allow Locus under Accessibility. - Linux uses wmctrl on X11 and says to install it when it's missing.
Reminders lived only in the running app, so closing Locus lost them. They're now saved in ~/.locus/reminders.json. When locus serve starts, it schedules them again and shows any that came due while Locus was closed, marked as missed. notify also gains list_reminders and cancel_reminder, and reminders can be up to 30 days out (was 24 hours).
Voice was push-to-talk only: click the mic, talk, click again, then send. With Hands-free voice on (Settings > Speech): - Ctrl+Shift+L (Cmd+Shift+L on a Mac) starts listening. It works in the window, and from any app in the desktop app, which raises Locus first. - A silence detector learns the room's noise for the first moments and stops recording when the speaker has been quiet for 1.2 s. If nobody speaks for 8 s it cancels; turns are capped at 30 s. - The transcript is sent like a typed message, and the reply is read aloud even when "Speak replies automatically" is off. - Pressing the shortcut again while a reply is being spoken interrupts it and listens; pressing it while listening sends straight away. The desktop app can now show, unminimize and focus its window so the global shortcut can bring it forward.
A Project button next to Agent picks a folder (a native folder picker in the desktop app). Chat then runs a developer agent kept inside that project: - file_read and file_write (with Undo) are limited to the project, and every file change shows a diff in the approval before it's written. - shell_exec runs in the project. Tests, linters and git status, diff, log and show run without asking; anything else, and anything with shell operators (; | > $( ) and similar), asks every time. - git tools default to the project and refuse paths outside it. Commits always ask. - The agent is told to plan, read before editing, work on a new branch instead of main, run the tests after changes and fix failures up to 3 times, and never push. The request field is mode "developer" with project_dir; a drive root, the home folder or a missing folder gets a 400 saying why. One agent is kept per project. Tools can now define risk_for before the name lists apply, so developer mode's shell can let known checks through. Anything marked requires_confirmation still always asks.
Built-in tools missing from DEFAULT_TOOL_CAPABILITIES are treated as system:admin and fail closed, and test_every_registered_builtin_has_an_ explicit_inventory_entry caught that the new tools weren't listed. task_plan is reviewed-safe (no capabilities, like think). open_path and app_control can start programs (code:execute), system_control changes computer settings (system:admin), clipboard and screenshot read and write data (file:read, file:write), notify schedules reminders (schedule:create), and window_control is tool:invoke.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Finishes
docs/design/2026-11-08-agent-mode-plan.md, so Agent mode can do tasks on the computer from text or voice. Each part is its own commit.Phase 2: attachments, images and web search reach the agent
AgentContext.metadata["input_images"].Phase 3 leftovers
task_plantool before multi-step work and as steps finish. The chat shows the latest plan as a checklist.file_writesaves the old contents under~/.locus/undo, and the tool card shows Undo. Undo only applies while the file still holds what the agent wrote, so later edits are never lost.tool:action. Rules are kept in~/.locus/agent_permissions.json, and Settings > Agent mode lists and removes them. Commands and code always ask.Phase 4: desktop control tools (seven tools in
locus.tools.desktop)open_path,app_control,window_control,system_control,clipboard,notifyandscreenshot.locus servestarts, and reminders missed while closed show right away.LOCUS_ARG_*environment variables.risk_for(arguments), so listing windows doesn't ask but closing an app does.Phase 5: hands-free voice (Settings > Speech)
Phase 6: developer mode
mode: "developer"withproject_dir.git status/diff/log/showrun without asking. Any other command, or any command with shell operators, asks every time. Commits always ask.Security inventory: the new tools are added to
DEFAULT_TOOL_CAPABILITIES, so they don't fail closed assystem:admin.Things a reviewer should know
default_denyon, theassistantanddeveloperagents get only the read baseline, so writing, commands and desktop control need explicit grants.core:window:allow-show,allow-unminimizeandallow-set-focusfor the hands-free shortcut.apply_patchin developer mode (it has no folder limit, so edits usefile_writewith a diff instead)How was this tested?
tests/tools/test_desktop_tools.py,test_undo_journal.pyandtest_task_plan.pytests/server/test_developer_mode.pytest_agent_mode.pyandtest_agent_approvals.pyfrontend/src/lib/vad.test.ts,agent-stop.test.tsandTaskPlan.test.tsToolCallCard.test.tstsc -bis clean.git statusandgit diffdidn't ask, a file edit asked with a diff and was applied, andrmasked and was deniedmain, every failure on this branch also fails onmain, mostly because thelocus_rustextension isn't installed locally. The one exception istests/tools/test_tool_timeout.py::test_repeated_timeouts_use_bounded_workers, which failed in the full run but passes alone, so it depends on test order.main: runningtests/toolsbeforetests/agentshangs attest_agent_routing_matrix.py::test_model_passthrough[llama3:70b-native_openhands]. That happens onmaintoo.Checklist
uv run pytest tests/ -v): new and changed suites pass; the remaining failures also fail onmainruff check src/and the new tests)