Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ Locus is that stack. It is a framework for local-first personal AI, built around
- **Attach files and images:** drop documents (PDF, Word, Excel, PowerPoint, OpenDocument, EPUB, HTML, CSV, code, plain text and more) or images into a chat and ask about them. Images need a vision model such as `qwen3.5` or `gemma3`.
- **Search the web:** turn on **Web search** and Locus looks up your question online first, reads the top pages, and answers with numbered links to its sources. It's off until you turn it on, since it sends your question to a search engine.
- **Work with your files:** read and write files and PDFs, search your own documents, run code in a sandbox, and remember facts between conversations.
- **Control your PC:** in Agent mode, open apps, files and folders, arrange windows, change the volume, brightness and theme, switch Wi-Fi and Bluetooth, lock the screen, use the clipboard, take screenshots, and set reminders. Locus shows its plan, asks in the chat before anything that changes something, and lets you undo file changes.
- **Hands-free voice:** press Ctrl+Shift+L (Cmd+Shift+L on a Mac), speak, and Locus answers aloud when you stop talking.
- **Developer mode:** point Agent mode at a project folder to read and edit code, run tests and fix failures, and work on a git branch, with a diff to approve before each change.
- **Connect your apps:** Gmail, Outlook, Google Calendar, Google Drive, Dropbox, Notion, Obsidian, GitHub, Slack and more, each only after you sign in.
- **Reply on messaging apps:** run it as a bot on Telegram, Slack, Discord, WhatsApp, Teams, Signal, email, SMS and about 25 other channels.
- **Do tasks on its own:** deep research reports, a daily morning digest, and agents that run on a schedule.
Expand Down
30 changes: 28 additions & 2 deletions docs/design/2026-11-08-agent-mode-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,8 @@ Most of the agent machinery is in the repo. The gap is mostly wiring and UX.

**Goal:** attachments, images and web search work in agent mode instead of silently falling back to plain chat.

> **Status (2026-11-08): implemented.** `_handle_agent` folds attached documents into each message's text (the same budget as direct chat), and the latest message's images reach the agent through `AgentContext.metadata["input_images"]`, which `BaseAgent._build_messages` puts on the final user message. Web results stay a pre-search, so the UI keeps its numbered sources: the web-context system message joins the agent's single system prompt. In Agent mode, requests with these extras no longer fall back to plain chat. Requests that don't use Agent mode route as before.

- Turn attachments into agent context: attached document text goes in as context or as a `file_read`-style handle; images go to vision models as today.
- Expose web search to the agent as its normal `web_search` tool when the toggle is on, rather than pre-injecting results. The agent decides when to search, and the privacy rule stays the same (off unless the user turns it on).
- Remove the `has_extras` bypass in `routes.py` once both paths are covered.
Expand All @@ -79,13 +81,16 @@ Most of the agent machinery is in the repo. The gap is mostly wiring and UX.

**Goal:** the user sees what the agent plans, what it's doing and what it needs approved, without leaving the chat.

> **Status (2026-11-08): approvals, risk levels and Stop are implemented. The plan checklist and Undo are not done yet.** As built:
> **Status (2026-11-08): implemented.** As built:
> - **Broker:** `src/locus/server/approvals.py`. The Agent mode agent's `before_tool_call` hook classifies each call as `read` (never asks), `change` (asks) or `run` (always asks; covers anything marked `requires_confirmation`). Unknown tools count as `change`, so new tools ask by default.
> - **Asking:** for a risky call, the agent's thread pauses, the stream sends `approval_required`, and the run waits up to 5 minutes for `POST /v1/chat/approvals/{id}` with `approve`, `approve_tool` (this tool for the rest of the run) or `deny`. No answer counts as a no. Requests that can't ask (non-streaming) deny risky calls.
> - **Stop:** each stream starts with an `agent_run` event. Stop calls `POST /v1/chat/runs/{run_id}/stop`, which ends the run at its next tool call (`AgentCancelled`), and the answer becomes "Stopped." Closing the stream does the same.
> - **Agent:** Agent mode now always uses its own `orchestrator`, not the server's configured agent, since only that agent has the hook. It adds `file_write`, `shell_exec`, `git_status`, `git_diff` and `git_log`, is kept to the home folder, runs tools one at a time (approvals are tracked per run on the agent's thread), and gets at least 25 turns.
> - **UI:** the tool card shows **Allow**, **Allow <tool> for this task** and **Deny**, with the command or path in words. The `assistant` preset stays read-only, because `locus serve` builds its agent without the hook.
> - **Not done yet:** a plan checklist before multi-step tasks, Undo for file changes, "always allow" rules that last beyond one run, and resuming a stopped run.
> - **Plan checklist:** a `task_plan` tool (never asks) that the agent calls before multi-step work and as steps finish. The chat shows the latest call as a checklist instead of a tool card.
> - **Undo:** Agent mode's `file_write` is `UndoableFileWriteTool` (`tools/undo_journal.py`). It saves the old contents under `~/.locus/undo` and returns an `undo_id`, the tool card shows Undo, and `POST /v1/chat/undo/{id}` puts the file back or deletes a file the agent created, only while it still holds what the agent wrote. Files over 10 MB aren't backed up; the last 100 changes are kept.
> - **"Always allow":** for change-level steps only, keyed by tool or `tool:action`, in `~/.locus/agent_permissions.json`. Settings > Agent mode lists and removes rules (`GET`/`DELETE /v1/chat/permissions`). Commands and code never get lasting rules. Approvals can also carry a `preview` from the tool's `preview_for()`, such as a file diff.
> - **Continue:** a stopped Agent mode reply says what was finished and shows Continue, which turns Agent mode on and asks the agent to pick up where it stopped.

- **Plan step:** for multi-step requests, the agent first emits a short plan (a new stream event type in `stream_bridge.py`); the UI shows it as a checklist that ticks off as steps finish.
- **Inline approvals:** when a tool needs approval, stream an `approval_required` event carrying the action from `ApprovalStore`; render Approve, Deny and "Always allow for this folder/command" buttons in the message. Keep `ApprovalBell` for background agents.
Expand All @@ -100,6 +105,23 @@ Most of the agent machinery is in the repo. The gap is mostly wiring and UX.

**Goal:** the agent can operate the computer, not only files and the browser.

> **Status (2026-11-08): implemented (Windows fully; macOS and Linux partly).** As built, in `src/locus/tools/desktop/`:
> - **Shared OS layer:** `_os.py` runs commands without a shell. Values from the model reach PowerShell only as `LOCUS_ARG_*` environment variables, or other commands as separate arguments, so a title or name can't inject a command. No new dependencies: Windows uses PowerShell, `ctypes` and `winreg`.
> - **Per-call risk:** tools can define `risk_for(arguments)`, so listing windows doesn't ask but closing an app does. `server/approvals.py` uses it; `requires_confirmation` still forces `run`.
>
> | Tool | Actions | Asks? | Windows | macOS | Linux |
> |---|---|---|---|---|---|
> | `open_path` | Open a file, folder or http(s) link | Files: no. Programs (`.exe`, `.ps1`, `.lnk`…): always. Links: yes | Yes | Yes | Yes (`xdg-open`) |
> | `app_control` | list, open, close (graceful, like clicking X) | list: no. open/close: yes | Yes | Yes | open only |
> | `window_control` | list, focus, minimize, maximize, restore (by title) | No | Yes | Not yet | Not yet |
> | `system_control` | get/set volume, mute, unmute, brightness, dark/light mode, lock screen | get_volume: no. Rest: yes | Yes (brightness on laptops) | No brightness | Volume (`pactl`), lock |
> | `clipboard` | read, write | Yes (reading can expose private text) | Yes | Yes | With `wl-clipboard` or `xclip` |
> | `notify` | Notification now, or a timer/reminder up to 24h | No | Yes (toast) | Yes | Yes (`notify-send`) |
> | `screenshot` | Whole screen to `~/.locus/screenshots` | Yes | Yes | Yes | With `gnome-screenshot` or `grim` |
>
> - **Added later the same day:** Wi-Fi and Bluetooth (`system_control` `get_radios`, `wifi_on/off`, `bluetooth_on/off`; Windows.Devices.Radios without admin rights, `networksetup` and `blueutil` on macOS, `nmcli` and `rfkill` on Linux). Window control on macOS (System Events; needs Accessibility permission) and Linux (`wmctrl`, X11 only). Reminders saved in `~/.locus/reminders.json` and restored when `locus serve` starts, with missed ones shown at once, plus `list_reminders` and `cancel_reminder`.
> - **Not done yet:** moving and snapping windows, window control on Wayland, and sending screenshots to vision models.

New tools under `src/locus/tools/desktop/`, one module per OS behind a common interface (Windows first, since that's where the desktop app is most used, then macOS, then Linux):

| Tool | Actions | Risk |
Expand All @@ -119,6 +141,8 @@ New tools under `src/locus/tools/desktop/`, one module per OS behind a common in

### Phase 5: hands-free voice loop

> **Status (2026-11-08): implemented, except a wake word.** Settings > Speech > Hands-free voice turns on Ctrl+Shift+L (Cmd+Shift+L on a Mac): in the window, and system-wide in the desktop app, which raises its window first (new `core:window` permissions). `lib/vad.ts` learns the noise floor, ends the turn after 1.2 s of quiet, cancels after 8 s of silence, and caps a turn at 30 s. The transcript is sent through the prompt queue and the reply is read aloud even with autoplay off. Pressing the shortcut during a spoken reply interrupts it and listens. Not built: a wake word, voice answers to approvals, and listening for the user talking over a reply (interrupting is by shortcut).

**Goal:** talk to Locus without touching the keyboard.

- **Activation:** a global hotkey (Tauri global shortcut) that opens the overlay and starts listening; an optional local wake word (for example openWakeWord) behind a setting, off by default.
Expand All @@ -132,6 +156,8 @@ New tools under `src/locus/tools/desktop/`, one module per OS behind a common in

### Phase 6: developer mode

> **Status (2026-11-08): implemented.** The Project button beside Agent picks a folder; requests then use `mode: "developer"` with `project_dir` (`server/developer.py`, one cached agent per folder). File, git and shell tools stay in the project, file edits show a diff and can be undone, and `command_risk` lets tests, linters and git status/diff/log/show run without asking while anything else (or anything with shell operators) asks every time. Commits always ask. The prompt asks for a branch before the first change, tests after changes (fixing failures up to 3 times), and no pushes. `apply_patch` isn't included (it has no folder limit); edits use `file_write` with a diff instead. No `developer` preset: it would run without the approval hook under `locus serve`.

**Goal:** a strong local coding agent built on the existing tools.

- A `developer` preset: `native_openhands` or `orchestrator` with `shell_exec`, `apply_patch`, `git_tool`, `file_read`/`file_write`, `code_interpreter_docker`, `http_request`, `db_query`, MCP.
Expand Down
3 changes: 3 additions & 0 deletions frontend/src-tauri/capabilities/default.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@
"notification:default",
"global-shortcut:allow-register",
"global-shortcut:allow-unregister",
"core:window:allow-show",
"core:window:allow-unminimize",
"core:window:allow-set-focus",
"autostart:allow-enable",
"autostart:allow-disable",
"autostart:allow-is-enabled",
Expand Down
88 changes: 88 additions & 0 deletions frontend/src/components/Chat/AgentPermissionsList.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
import { useCallback, useEffect, useState } from 'react';
import { toast } from 'sonner';
import { fetchAgentPermissions, removeAgentPermission, type AgentPermission } from '../../lib/api';

const ACTION_WORDS: Record<string, string> = {
set_volume: 'set the volume',
mute: 'mute',
unmute: 'unmute',
set_brightness: 'set the brightness',
dark_mode: 'switch to dark mode',
light_mode: 'switch to light mode',
lock_screen: 'lock the screen',
wifi_on: 'turn Wi-Fi on',
wifi_off: 'turn Wi-Fi off',
bluetooth_on: 'turn Bluetooth on',
bluetooth_off: 'turn Bluetooth off',
open: 'open',
close: 'close',
read: 'read',
write: 'write',
};

/** A rule in words, like "system_control: set the volume". */
export function describeRule(rule: AgentPermission): string {
if (!rule.action) return `${rule.tool}: any use`;
return `${rule.tool}: ${ACTION_WORDS[rule.action] ?? rule.action}`;
}

/** Agent mode's "always allow" rules, each with a Remove button. */
export function AgentPermissionsList() {
const [rules, setRules] = useState<AgentPermission[] | null>(null);
const [error, setError] = useState<string | null>(null);

const load = useCallback(() => {
fetchAgentPermissions()
.then((r) => {
setRules(r);
setError(null);
})
.catch((err) => setError(err?.message || 'Could not load the rules.'));
}, []);

useEffect(load, [load]);

const remove = async (key: string) => {
try {
await removeAgentPermission(key);
setRules((prev) => (prev ?? []).filter((r) => r.key !== key));
} catch (err: any) {
toast.error(err?.message || 'Could not remove the rule.');
}
};

if (error) {
return <div className="text-xs" style={{ color: 'var(--color-text-tertiary)' }}>{error}</div>;
}
if (rules === null) {
return <div className="text-xs" style={{ color: 'var(--color-text-tertiary)' }}>Loading…</div>;
}
if (rules.length === 0) {
return (
<div className="text-xs" style={{ color: 'var(--color-text-tertiary)' }}>
No steps are always allowed. Choose Always allow on an approval to add one. Commands and code always ask.
</div>
);
}
return (
<ul className="flex flex-col">
{rules.map((rule) => (
<li
key={rule.key}
className="flex items-center justify-between py-2 text-sm"
style={{ borderBottom: '1px solid var(--color-border-subtle)', color: 'var(--color-text)' }}
>
<span>{describeRule(rule)}</span>
<button
type="button"
onClick={() => remove(rule.key)}
className="px-2 py-0.5 rounded text-xs cursor-pointer"
style={{ border: '1px solid var(--color-border)', color: 'var(--color-text-secondary)' }}
>
Remove
</button>
</li>
))}
</ul>
);
}
7 changes: 5 additions & 2 deletions frontend/src/components/Chat/ChatArea.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,10 @@ export function ChatArea() {
streamingConversationRef.current = isCurrentChatStreaming ? activeId : null;

if (!justFinished) return;
if (!voiceOutputEnabled || !voiceAutoplay) return;
// A hands-free turn always hears its reply; otherwise follow the settings.
const handsFreeTurn = useAppStore.getState().speakNextReply;
if (handsFreeTurn) useAppStore.getState().setSpeakNextReply(false);
if (!handsFreeTurn && (!voiceOutputEnabled || !voiceAutoplay)) return;

if (!last || last.role !== 'assistant' || activeId === null) return;

Expand All @@ -75,7 +78,7 @@ export function ChatArea() {
const currentTts = useTtsStore.getState();
const currentLast = app.messages[app.messages.length - 1];
if (app.activeId !== completedConversationId || app.streamState.isStreaming) return;
if (!app.settings.voiceOutputEnabled || !app.settings.voiceAutoplay) return;
if (!handsFreeTurn && (!app.settings.voiceOutputEnabled || !app.settings.voiceAutoplay)) return;
if (currentLast?.id !== completedMessageId || currentTts.available !== true) return;
if (currentTts.autoSpokenId === completedMessageId) return;

Expand Down
Loading
Loading