From 518e0e5c607a4b247762c17ab184af472a710918 Mon Sep 17 00:00:00 2001 From: Aryan Date: Sun, 8 Nov 2026 18:20:00 +0530 Subject: [PATCH 1/2] docs(design): plan for turning chat into a task-doing agent Six phases: agent mode in chat, attachments and web search through the agent, plans and approvals in the conversation, desktop control tools, a hands-free voice loop, and developer mode. Lists what already exists in the repo, the gaps, and what "done" means for each phase. --- docs/design/2026-11-08-agent-mode-plan.md | 162 ++++++++++++++++++++++ 1 file changed, 162 insertions(+) create mode 100644 docs/design/2026-11-08-agent-mode-plan.md diff --git a/docs/design/2026-11-08-agent-mode-plan.md b/docs/design/2026-11-08-agent-mode-plan.md new file mode 100644 index 0000000..fcdd7d8 --- /dev/null +++ b/docs/design/2026-11-08-agent-mode-plan.md @@ -0,0 +1,162 @@ +# Agent mode: turning Locus chat into a task-doing agent + +| | | +|---|---| +| **Date** | 2026-11-08 | +| **Status** | Plan, not started | +| **Owner** | Locus team | +| **Scope** | Desktop app, browser GUI and terminal chat | + +> **For agents picking this up cold:** read §1 and §2 first. Each phase in §4 lists the files it touches and what "done" means, and phases 1–3 must land in order. Phases 4, 5 and 6 are independent of each other once phase 3 is in. + +## 1. Goal + +Today Locus chat answers questions. The goal is for the same chat box, by text or voice, to **carry out tasks** on the user's machine: open and organize files, run commands, drive the browser, control the desktop, draft and send email after approval, and work through multi-step developer tasks. Everything stays local by default and every risky step needs the user's approval. + +**Success looks like:** a user says "Find last month's invoices in Downloads, move them into Documents/Invoices and summarize the totals." Locus shows a short plan, runs the steps with visible progress, asks before moving files, and answers aloud with the result. + +## 2. What already exists + +Most of the agent machinery is in the repo. The gap is mostly wiring and UX. + +| Area | What's there | Where | +|---|---|---| +| Agent loops | `orchestrator`, `native_react`, `native_openhands` (CodeAct), `operative`, `deep_research`, `monitor_operative`, all with `max_turns` | `src/locus/agents/` | +| Tools | ~40 tools: `shell_exec`, `file_read`/`file_write`, `apply_patch`, `git_tool`, `browser` + `browser_axtree`, `code_interpreter` (+ Docker), `web_search`, `http_request`, `memory_manage`, `pdf_tool`, `apple_calendar`, `mcp_adapter`, `text_to_speech` | `src/locus/tools/` | +| Agent streaming | `AgentStreamBridge` streams agent events over SSE | `src/locus/server/stream_bridge.py` | +| Approvals | `ApprovalStore` (pending actions + permission rules), REST routes, `ApprovalBell` UI | `src/locus/tools/approval_store.py`, `src/locus/server/approval_routes.py`, `frontend/src/components/ApprovalBell.tsx` | +| Security | Capability policy, guardrails, injection scanner, subprocess sandbox, SSRF guard, audit log, taint tracking | `src/locus/security/` | +| Speech to text | faster-whisper, OpenAI Whisper, Deepgram; push-to-talk in UI | `src/locus/speech/`, `frontend/src/hooks/useSpeech.ts` | +| Text to speech | Kokoro (local), ElevenLabs, Cartesia, OpenAI | `src/locus/speech/` | +| Tool call UI | Tool call cards in chat, trace debugger | `frontend/src/components/Chat/ToolCallCard.tsx`, `.../TraceDebugger.tsx` | +| Quick overlay | Small chat overlay window | `desktop/src-tauri/src/overlay.html` | +| Model sizing | RAM-based model ladder (`qwen3.5:0.8b` → `122b`), startup model `qwen3.5:4b`, fallback `qwen3:0.6b` | `frontend/src-tauri/src/lib.rs` | + +## 3. Gaps that stop it acting as an agent + +1. **The desktop chat runs the no-tools agent.** The desktop app starts the server with `--agent simple` (`frontend/src-tauri/src/lib.rs`), so its chat never calls tools. (`locus serve` on its own defaults to `orchestrator` with `think`, `calculator` and `web_search`.) +2. **Rich requests bypass the agent.** `chat_completions` in `src/locus/server/routes.py` sends any request with images, attachments or web search straight to the engine, because agents take only the latest message's text. Non-streaming requests use the agent only when it has tools. +3. **No in-chat task flow.** Approvals live in a bell, not in the conversation. There's no visible plan, step progress, cancel or resume. +4. **No desktop control.** No tools to open apps, manage windows, change volume or brightness, use the clipboard, show notifications or take screenshots. Only the browser can be driven. +5. **Voice is push-to-talk only.** No hotkey or wake word, no end-of-speech detection, replies aren't spoken automatically, and the user can't interrupt. +6. **Small models are unreliable at tool calls.** The RAM ladder picks 0.8b–4b on 4–8 GB machines; those miss or garble tool calls on multi-step tasks. + +## 4. Phases + +### Phase 1: agent mode in chat (foundation) + +**Goal:** chat can call tools, with a switch between "Chat" and "Agent". + +> **Status (2026-11-08): implemented.** As built: +> - The request field is `mode: "chat" | "agent"`. Leaving it unset keeps the old routing, so `locus gui` users on the `orchestrator` default see no change. +> - `src/locus/server/task_agent.py` reuses the configured agent if it has tools. Otherwise it builds one `orchestrator` the first time it's needed and caches it on `app.state.task_agent`. +> - The tool set is `think`, `calculator`, `file_read`, `pdf_extract` and `memory_manage`. `file_read` is kept to the home folder unless `security.allowed_dirs` is set. `file_write` and `shell_exec` wait for phase 3, because the server has no way to ask for approval yet (`shell_exec` requires confirmation and is refused there). +> - Prior messages already reached agents through `AgentContext.conversation`, so no change was needed there. +> - Agent streams now forward that agent's `agent_turn_start`, `tool_call_start` and `tool_call_end` events live. Events from other agents on the bus are filtered out. +> - The small-model notice is a check on the model's name in the frontend (`isSmallForAgent`: under 4B). +> - With web search or attachments in the conversation, the UI says Agent mode can't use them yet and gives a chat answer. Phase 2 removes that limit. + +- Add an `agent` mode to the chat request (`ChatCompletionRequest` in `src/locus/server/models.py`), defaulting to the configured agent. Keep `simple` available as "Chat" mode. +- Add a starter preset `assistant` (`orchestrator` + a safe default tool set: `file_read`, `file_write`, `shell_exec`, `web_search`, `browser`, `memory_manage`, `calculator`, `think`). Make it the desktop app's default on first launch; existing users keep their config. +- Build the agent per request (or per session) instead of only once at server start, so the UI's mode switch takes effect without a restart. Check `app.state.agent` usage in `routes.py` and `agent_manager_routes.py`. +- Give the agent the full conversation, not only the last message. +- **Frontend:** add a Chat/Agent toggle to `InputArea.tsx`; persist it per conversation. +- **Model check:** before running in agent mode, check whether the model supports tool calling (extend `src/locus/server/model_capabilities.py`). If it doesn't, or it's under ~4B, show a one-line notice suggesting a bigger model. + +**Done when:** in agent mode, "what files are in my Downloads folder?" calls `file_read`/`shell_exec` and the tool call card appears; chat mode behaves exactly as today; tests cover both paths. + +### Phase 2: attachments and web search through the agent + +**Goal:** attachments, images and web search work in agent mode instead of silently falling back to plain chat. + +- Turn attachments into agent context: attached document text goes in as context or as a `file_read`-style handle; images go to vision models as today. +- Expose web search to the agent as its normal `web_search` tool when the toggle is on, rather than pre-injecting results. The agent decides when to search, and the privacy rule stays the same (off unless the user turns it on). +- Remove the `has_extras` bypass in `routes.py` once both paths are covered. + +**Done when:** "summarize this PDF and email the summary to me" works with an attachment in agent mode; the existing attachment and web-search tests pass in both modes. + +### Phase 3: task flow and approvals in the conversation + +**Goal:** the user sees what the agent plans, what it's doing and what it needs approved, without leaving the chat. + +- **Plan step:** for multi-step requests, the agent first emits a short plan (a new stream event type in `stream_bridge.py`); the UI shows it as a checklist that ticks off as steps finish. +- **Inline approvals:** when a tool needs approval, stream an `approval_required` event carrying the action from `ApprovalStore`; render Approve, Deny and "Always allow for this folder/command" buttons in the message. Keep `ApprovalBell` for background agents. +- **Risk levels:** classify tools as read-only (no prompt), reversible write (prompt once per session or per folder) or irreversible/outward (always prompt: delete, send, purchase, system settings). Reuse `severity_policy.py` and `capabilities.py`. +- **Cancel and resume:** a Stop button that cancels the running agent cleanly; the session store keeps enough state to continue with "keep going". +- **Undo:** for file moves and renames, record the inverse operation and offer Undo in the result. +- **Raise `max_turns`** for agent mode (for example 25), with the loop guard (`loop_guard.py`) catching repeated calls. + +**Done when:** the invoice example from §1 runs end to end with a visible plan, one approval prompt and a working Undo; Stop interrupts within a second; risky tools can't run without approval (tested). + +### Phase 4: desktop control tools + +**Goal:** the agent can operate the computer, not only files and the browser. + +New tools under `src/locus/tools/desktop/`, one module per OS behind a common interface (Windows first, since that's where the desktop app is most used, then macOS, then Linux): + +| Tool | Actions | Risk | +|---|---|---| +| `app_control` | Open, focus, close apps; list running apps | Reversible (close = prompt) | +| `window_control` | List, move, resize, minimize, snap windows | Read-only / reversible | +| `system_control` | Volume, mute, brightness, dark mode, Wi-Fi/Bluetooth on or off, lock screen | Prompt every time | +| `clipboard` | Read, write | Read = prompt once per session | +| `notify` | Desktop notification, timer, alarm, reminder | No prompt | +| `screen` | Screenshot (for vision models), read text on screen | Prompt once per session | +| `open_path` | Open a file, folder or URL in its default app | No prompt for files, prompt for executables | + +- Implementation candidates: `pywinauto`/`pycaw`/Win32 APIs on Windows, AppleScript/`osascript` on macOS, `wmctrl`/`xdotool`/D-Bus on Linux. Put them in a `desktop-control` optional extra. +- Full mouse-and-keyboard UI automation is deliberately **out of scope** for this phase: it's fragile and risky. Revisit after the structured tools are in. + +**Done when:** "turn the volume down to 30 percent and open my Downloads folder" works on Windows; each tool has unit tests with the OS layer mocked; risky actions go through phase 3 approvals. + +### Phase 5: hands-free voice loop + +**Goal:** talk to Locus without touching the keyboard. + +- **Activation:** a global hotkey (Tauri global shortcut) that opens the overlay and starts listening; an optional local wake word (for example openWakeWord) behind a setting, off by default. +- **End-of-speech detection:** voice activity detection (Silero VAD or WebRTC VAD) so recording stops when the user stops talking. +- **Spoken replies:** in voice mode, speak the final answer with Kokoro (local) by default. Speak short status lines for long tasks ("Moving 12 files…"). +- **Interruption:** when the user starts talking during playback, stop playback and start listening. +- **Voice approvals:** "yes", "no" and "cancel" answer an approval prompt, but irreversible actions (delete, send, purchase) also need a click or a typed confirmation. +- Show the transcript as editable text before sending, with a setting to auto-send. + +**Done when:** from the hotkey, "what's on my calendar today?" is answered aloud with no clicks; interrupting stops playback within ~300 ms; everything runs offline with faster-whisper + Kokoro. + +### Phase 6: developer mode + +**Goal:** a strong local coding agent built on the existing tools. + +- A `developer` preset: `native_openhands` or `orchestrator` with `shell_exec`, `apply_patch`, `git_tool`, `file_read`/`file_write`, `code_interpreter_docker`, `http_request`, `db_query`, MCP. +- **Project scope:** the user picks a project folder; file and shell tools are limited to it (`file_policy.py`, `mount_security.py`). +- **Diff review:** show `apply_patch` changes as a diff to accept or reject before writing. +- **Test and fix loop:** run the project's test command, read failures, patch, re-run, up to a turn limit. +- **Command allowlist:** safe commands (`git status`, test runners, linters) run without prompts; anything else asks. +- **Git hygiene:** work on a new branch, write commit messages, never push without approval. + +**Done when:** on a sample repo with a failing test, "fix the failing test" produces a reviewed diff, a passing test run and a local commit on a new branch. + +## 5. Order and dependencies + +``` +Phase 1 ──► Phase 2 ──► Phase 3 ──┬──► Phase 4 (desktop tools) + ├──► Phase 5 (voice loop) + └──► Phase 6 (developer mode) +``` + +Phases 4–6 can run in parallel once phase 3 is merged, since they touch separate files. + +## 6. Cross-cutting requirements + +- **Privacy:** no new network calls by default. Wake word, speech to text, text to speech and every new tool run locally. +- **Safety:** every tool declares a risk level; irreversible and outward actions always prompt; all tool calls go to the audit log and traces. +- **Prompt injection:** tool output (web pages, files, email) is treated as data; keep `injection_scanner.py` on that output and never let it approve actions. +- **Small machines:** on machines that only fit models under ~4B, agent mode still works but warns that results may be unreliable, and suggests a larger model or a cloud key. +- **Tests:** each phase adds unit tests and at least one end-to-end test with a scripted fake engine, so CI doesn't need a real model. +- **Docs:** user guide pages for agent mode, voice mode and developer mode; update the README "What Locus can do" section. + +## 7. Open questions + +1. Should agent mode become the default for new desktop installs, or stay opt-in for the first release? +2. Which minimum model should agent mode recommend on 8 GB machines (`qwen3.5:4b` vs a tool-tuned alternative)? Needs a small benchmark with `locus bench`. +3. Wake word: ship it in the first voice release or after? +4. Should approvals be shareable across channels (approve a desktop action from Telegram), reusing the proactive agent's approval routing? From f17a5245e1d24a5ae8fef3fd759c0836f1a27431 Mon Sep 17 00:00:00 2001 From: Aryan Date: Sun, 8 Nov 2026 18:20:02 +0530 Subject: [PATCH 2/2] feat(chat): add an Agent mode that uses tools The desktop app serves with --agent simple, so its chat never called tools. Add an Agent toggle next to Web search and Deep Research. - Chat requests take an optional mode, "chat" or "agent". Unset keeps the existing routing, so the server's configured agent still answers. - Agent mode reuses the configured agent when it has tools; otherwise it builds an orchestrator on first use with read-only tools (think, calculator, file_read, pdf_extract, memory_manage). file_read stays in the home folder unless security.allowed_dirs is set. Writing files and running commands wait for approvals in the chat. - Agent streams forward that agent's turn and tool events as they happen, so tool cards fill in live. Events from other agents on the bus are skipped. - Warn when the selected model is under 4B, and give a chat answer when the conversation has web search or attachments, which the agent can't use yet. - Add an "assistant" preset with the same tools. --- README.md | 1 + configs/locus/examples/assistant.toml | 32 +++ frontend/src/components/Chat/InputArea.tsx | 52 +++- frontend/src/lib/model-capabilities.test.ts | 31 ++- frontend/src/lib/model-capabilities.ts | 13 + frontend/src/lib/sse.ts | 2 + frontend/src/lib/store.ts | 12 + src/locus/cli/init_cmd.py | 1 + src/locus/server/models.py | 5 +- src/locus/server/routes.py | 76 +++++- src/locus/server/task_agent.py | 137 +++++++++++ tests/server/test_agent_mode.py | 249 ++++++++++++++++++++ 12 files changed, 606 insertions(+), 5 deletions(-) create mode 100644 configs/locus/examples/assistant.toml create mode 100644 src/locus/server/task_agent.py create mode 100644 tests/server/test_agent_mode.py diff --git a/README.md b/README.md index 5b251ac..91827d8 100644 --- a/README.md +++ b/README.md @@ -133,6 +133,7 @@ uv run pytest tests/ -v | `code-assistant` | Agent with code execution, file I/O, and shell access | | `scheduled-monitor` | Stateful agent on a schedule with memory | | `chat-simple` | Lightweight conversation, no tools | +| `assistant` | Agent that reads files, PDFs and its memory to do tasks | Example: diff --git a/configs/locus/examples/assistant.toml b/configs/locus/examples/assistant.toml new file mode 100644 index 0000000..df06193 --- /dev/null +++ b/configs/locus/examples/assistant.toml @@ -0,0 +1,32 @@ +# Assistant — an agent that uses tools to do tasks, not only chat +# Copy to ~/.locus/config.toml +# +# The same tool set the desktop app's Agent mode uses. Every tool here only +# reads, or writes to Locus's own memory: writing files, running commands and +# network access are left out until they can ask you in the chat first. +# +# Usage: +# locus ask "How many PDFs are in my Downloads folder?" +# locus serve # chat requests use this agent by default + +[engine] +default = "ollama" + +[intelligence] +default_model = "qwen3.5:9b" # Small models often miss tool calls +# default_model = "qwen3.5:4b" # For 8 GB machines; less reliable + +[agent] +default_agent = "orchestrator" # Multi-turn with tool selection +max_turns = 10 + +[server] +agent = "orchestrator" +host = "127.0.0.1" +port = 8000 + +[tools] +enabled = ["think", "calculator", "file_read", "pdf_extract", "memory_manage"] + +[security] +allowed_dirs = ["~"] # File tools stay in your home folder diff --git a/frontend/src/components/Chat/InputArea.tsx b/frontend/src/components/Chat/InputArea.tsx index 541f5c0..7a01c8b 100644 --- a/frontend/src/components/Chat/InputArea.tsx +++ b/frontend/src/components/Chat/InputArea.tsx @@ -1,11 +1,12 @@ import { useState, useRef, useCallback, useEffect } from 'react'; -import { Send, Square, Paperclip, Search, Globe } from 'lucide-react'; +import { Send, Square, Paperclip, Search, Globe, Bot } from 'lucide-react'; import { toast } from 'sonner'; import { useAppStore, generateId } from '../../lib/store'; import { streamChat, streamResearch } from '../../lib/sse'; import { fetchSavings, getBase } from '../../lib/api'; import { listConnectors, getSyncStatus } from '../../lib/connectors-api'; import { serializeToolCallArguments } from '../../lib/tool-call'; +import { isSmallForAgent } from '../../lib/model-capabilities'; import { engineFromCompletionChunk, resolveChatEngine, @@ -122,6 +123,8 @@ export function InputArea() { const webSearch = useAppStore((s) => s.webSearch); const setWebSearch = useAppStore((s) => s.setWebSearch); const setLastAssistantWebSearch = useAppStore((s) => s.setLastAssistantWebSearch); + const agentMode = useAppStore((s) => s.agentMode); + const setAgentMode = useAppStore((s) => s.setAgentMode); const corpusSync = useResearchCorpusSync(deepResearch); const [pending, setPending] = useState([]); @@ -188,6 +191,12 @@ export function InputArea() { const toggleDeepResearch = () => { setDeepResearch(!deepResearch); if (!deepResearch && webSearch) setWebSearch(false); + if (!deepResearch && agentMode) setAgentMode(false); + }; + // Agent mode and Deep Research both decide how to answer, so only one is on. + const toggleAgentMode = () => { + setAgentMode(!agentMode); + if (!agentMode && deepResearch) setDeepResearch(false); }; const isCurrentChatStreaming = streamState.isStreaming && streamState.conversationId === activeId; @@ -304,6 +313,17 @@ export function InputArea() { ...(m.role === 'user' ? await apiAttachmentFields(m.attachments) : {}), })), ); + // The agent can't use web results or attached files yet, so a + // conversation that has either gets a chat answer. + const hasExtras = + searchWeb || + apiMessages.some( + (m) => ('images' in m && m.images?.length) || ('attachments' in m && m.attachments?.length), + ); + const useAgent = agentMode && !deepResearch && !hasExtras; + if (agentMode && !deepResearch && hasExtras) { + toast.info("Agent mode can't use web search or attached files yet, so this message gets a chat answer."); + } const assistantMsg: ChatMessage = { id: generateId(), @@ -487,6 +507,7 @@ export function InputArea() { temperature, max_tokens: maxTokens, ...(searchWeb ? { web_search: true } : {}), + ...(useAgent ? { mode: 'agent' as const } : {}), }, controller.signal, )) { @@ -692,6 +713,7 @@ export function InputArea() { attachmentsReading, webSearch, setLastAssistantWebSearch, + agentMode, ]); const handleKeyDown = (e: React.KeyboardEvent) => { @@ -741,7 +763,35 @@ export function InputArea() { Deep Research + + {agentMode && selectedModel && isSmallForAgent(selectedModel) && ( +
+ Small models like {selectedModel} often miss tool calls. A 4B or larger model works better in Agent mode. +
+ )} {deepResearch && corpusSync.syncing && corpusSync.itemsSynced > 0 && (
{ it.each([ @@ -17,3 +17,32 @@ describe('isEmbedOnlyModel', () => { expect(isEmbedOnlyModel(modelId)).toBe(false); }); }); + +describe('modelSizeBillions', () => { + it.each([ + ['qwen3.5:4b', 4], + ['qwen3:0.6b', 0.6], + ['llama3.1:8b-instruct-q4_K_M', 8], + ['qwen3.5:35b-a3b', 35], + ['hf.co/Qwen/Qwen3-1.7B-GGUF', 1.7], + ])('reads %s as %s billion', (modelId, size) => { + expect(modelSizeBillions(modelId)).toBe(size); + }); + + it.each(['gpt-4o', 'gemma3n:e4b', 'llama3.1:latest'])('returns null for %s', (modelId) => { + expect(modelSizeBillions(modelId)).toBeNull(); + }); +}); + +describe('isSmallForAgent', () => { + it('flags models under 4B', () => { + expect(isSmallForAgent('qwen3.5:2b')).toBe(true); + expect(isSmallForAgent('qwen3:0.6b')).toBe(true); + }); + + it('allows 4B and up, and models without a size', () => { + expect(isSmallForAgent('qwen3.5:4b')).toBe(false); + expect(isSmallForAgent('qwen3.5:9b')).toBe(false); + expect(isSmallForAgent('gpt-4o')).toBe(false); + }); +}); diff --git a/frontend/src/lib/model-capabilities.ts b/frontend/src/lib/model-capabilities.ts index 49e651b..a52b52a 100644 --- a/frontend/src/lib/model-capabilities.ts +++ b/frontend/src/lib/model-capabilities.ts @@ -20,3 +20,16 @@ export function isEmbedOnlyModel(modelId: string): boolean { EMBEDDING_MODEL_PREFIXES.some((prefix) => leaf.startsWith(prefix)) ); } + +/** Parameter count in billions from a model id like `qwen3.5:4b`, if it names one. */ +export function modelSizeBillions(modelId: string): number | null { + const name = (modelId || '').trim().toLowerCase(); + const match = name.match(/(?:^|[:\-_/])(\d+(?:\.\d+)?)b(?=$|[-_.:])/); + return match ? Number(match[1]) : null; +} + +/** Models below about 4B parameters often miss or garble tool calls. */ +export function isSmallForAgent(modelId: string): boolean { + const size = modelSizeBillions(modelId); + return size !== null && size < 4; +} diff --git a/frontend/src/lib/sse.ts b/frontend/src/lib/sse.ts index e56a455..84ece45 100644 --- a/frontend/src/lib/sse.ts +++ b/frontend/src/lib/sse.ts @@ -16,6 +16,8 @@ export interface ChatRequest { max_tokens?: number; /** Search the web for the latest user message before answering. */ web_search?: boolean; + /** "agent" runs a tool-using agent; "chat" answers with the model alone. */ + mode?: 'chat' | 'agent'; } export async function* streamChat( diff --git a/frontend/src/lib/store.ts b/frontend/src/lib/store.ts index ce8e7ea..eaeb42a 100644 --- a/frontend/src/lib/store.ts +++ b/frontend/src/lib/store.ts @@ -44,6 +44,7 @@ const OPTIN_NAME_KEY = 'locus-display-name'; const OPTIN_EMAIL_KEY = 'locus-email'; const OPTIN_ANONID_KEY = 'locus-anon-id'; const WEB_SEARCH_KEY = 'locus-web-search'; +const AGENT_MODE_KEY = 'locus-agent-mode'; interface ConversationStore { version: 1; @@ -214,6 +215,10 @@ interface AppState { // Web search toggle (remembered across launches) and what an answer searched webSearch: boolean; setWebSearch: (on: boolean) => void; + + // Agent mode: answers use a tool-using agent (remembered across launches) + agentMode: boolean; + setAgentMode: (on: boolean) => void; setLastAssistantWebSearch: ( conversationId: string, webSearch: WebSearchInfo, @@ -502,6 +507,13 @@ export const useAppStore = create((set, get) => { localStorage.setItem(WEB_SEARCH_KEY, String(on)); set({ webSearch: on }); }, + + // ── Agent mode ──────────────────────────────────────────────── + agentMode: localStorage.getItem(AGENT_MODE_KEY) === 'true', + setAgentMode: (on: boolean) => { + localStorage.setItem(AGENT_MODE_KEY, String(on)); + set({ agentMode: on }); + }, setLastAssistantWebSearch: (conversationId, webSearch, sources) => { const store = loadConversations(); const conv = store.conversations[conversationId]; diff --git a/src/locus/cli/init_cmd.py b/src/locus/cli/init_cmd.py index f02af28..b25b7f7 100644 --- a/src/locus/cli/init_cmd.py +++ b/src/locus/cli/init_cmd.py @@ -284,6 +284,7 @@ def _do_download(engine: str, model: str, spec, console: Console) -> None: "code-assistant", "scheduled-monitor", "chat-simple", + "assistant", ], case_sensitive=False, ), diff --git a/src/locus/server/models.py b/src/locus/server/models.py index b080b2c..5703865 100644 --- a/src/locus/server/models.py +++ b/src/locus/server/models.py @@ -4,7 +4,7 @@ import time import uuid -from typing import Any, Dict, List, Optional +from typing import Any, Dict, List, Literal, Optional from pydantic import BaseModel, Field @@ -44,6 +44,9 @@ class ChatCompletionRequest(BaseModel): tools: Optional[List[Dict[str, Any]]] = None # Locus extension: search the web for the latest user message first. web_search: bool = False + # Locus extension: "chat" answers with the model alone, "agent" runs a + # tool-using agent. Unset keeps the server's configured agent. + mode: Optional[Literal["chat", "agent"]] = None # --------------------------------------------------------------------------- diff --git a/src/locus/server/routes.py b/src/locus/server/routes.py index a64efc6..9a2c217 100644 --- a/src/locus/server/routes.py +++ b/src/locus/server/routes.py @@ -259,6 +259,12 @@ async def chat_completions(request_body: ChatCompletionRequest, request: Request """Handle chat completion requests (streaming and non-streaming).""" engine = request.app.state.engine agent = getattr(request.app.state, "agent", None) + if request_body.mode == "chat": + agent = None + elif request_body.mode == "agent": + from locus.server.task_agent import get_task_agent + + agent = await asyncio.to_thread(get_task_agent, request.app.state) model = request_body.model # Agents take only the latest message's text, which would drop web results, # attached documents and images, so those requests go straight to the engine. @@ -808,6 +814,57 @@ def _handle_agent( ) +def _forward_agent_events(bus, agent, loop, queue: asyncio.Queue): + """Queue *agent*'s turn and tool events from *bus* as SSE strings. + + Events from other agents sharing the bus (scheduled or managed agents) + are skipped. Returns a function that removes the subscriptions. + """ + if bus is None: + return lambda: None + + import json + + from locus.core.events import EventType + from locus.server.task_agent import agent_event_ids + + ids = agent_event_ids(agent) + names = { + EventType.AGENT_TURN_START: "agent_turn_start", + EventType.TOOL_CALL_START: "tool_call_start", + EventType.TOOL_CALL_END: "tool_call_end", + } + callbacks = {} + + def make_callback(name): + def callback(event): + data = event.data or {} + if data.get("agent") not in ids: + return + if name == "agent_turn_start": + data = {"agent": data.get("agent")} + elif name == "tool_call_start" and not isinstance( + data.get("arguments"), str + ): + # The UI expects arguments as JSON text, as the bus carries + # them parsed for trace consumers. + data = {**data, "arguments": json.dumps(data.get("arguments"))} + sse = f"event: {name}\ndata: {json.dumps(data, default=str)}\n\n" + loop.call_soon_threadsafe(queue.put_nowait, sse) + + return callback + + for event_type, name in names.items(): + callbacks[event_type] = make_callback(name) + bus.subscribe(event_type, callbacks[event_type]) + + def unsubscribe(): + for event_type, callback in callbacks.items(): + bus.unsubscribe(event_type, callback) + + return unsubscribe + + async def _handle_agent_stream( agent, model: str, @@ -844,8 +901,13 @@ async def generate(): ) yield f"data: {first_chunk.model_dump_json()}\n\n" - try: - response = await asyncio.to_thread( + # Forward this agent's turn and tool events while it runs, so the UI + # can show each tool call as it happens rather than only the answer. + loop = asyncio.get_running_loop() + live_events: asyncio.Queue = asyncio.Queue() + unsubscribe = _forward_agent_events(bus, agent, loop, live_events) + task = asyncio.ensure_future( + asyncio.to_thread( _handle_agent, agent, model, @@ -854,6 +916,14 @@ async def generate(): trace_store=trace_store, bus=bus, ) + ) + task.add_done_callback( + lambda _f: loop.call_soon_threadsafe(live_events.put_nowait, None) + ) + try: + while (event := await live_events.get()) is not None: + yield event + response = task.result() except Exception as exc: logging.getLogger("locus.server").error( "Agent stream error: %s", @@ -875,6 +945,8 @@ async def generate(): yield f"data: {error_chunk.model_dump_json()}\n\n" yield "data: [DONE]\n\n" return + finally: + unsubscribe() content = _response_content(response) if content: diff --git a/src/locus/server/task_agent.py b/src/locus/server/task_agent.py new file mode 100644 index 0000000..03bff0f --- /dev/null +++ b/src/locus/server/task_agent.py @@ -0,0 +1,137 @@ +"""The tool-using agent behind the chat's Agent mode. + +When the server's configured agent already has tools, Agent mode uses it. +Otherwise (the desktop app serves with ``--agent simple``) a separate +``orchestrator`` is built on first use with :data:`ASSISTANT_TOOLS` and cached +on ``app.state``. +""" + +from __future__ import annotations + +import logging +import threading +from pathlib import Path +from typing import Any, Optional + +logger = logging.getLogger("locus.server") + +# Tools that can't change anything outside Locus's own memory. Writing files, +# running commands and network access wait for in-chat approvals. +ASSISTANT_TOOLS = ( + "think", + "calculator", + "file_read", + "pdf_extract", + "memory_manage", +) + +ASSISTANT_AGENT_ID = "assistant" + +_build_lock = threading.Lock() + + +def has_tools(agent: Any) -> bool: + return agent is not None and bool(getattr(agent, "_tools", None)) + + +def get_task_agent(state: Any) -> Optional[Any]: + """Return a tool-using agent for *state* (``app.state``), or ``None``.""" + configured = getattr(state, "agent", None) + if has_tools(configured): + return configured + cached = getattr(state, "task_agent", None) + if cached is not None: + return cached + with _build_lock: + cached = getattr(state, "task_agent", None) + if cached is None: + try: + cached = _build_task_agent(state) + except Exception: + logger.warning("Agent mode could not start", exc_info=True) + return None + state.task_agent = cached + return cached + + +def _assistant_tools(config: Any) -> list: + import locus.tools # noqa: F401 # trigger registration + from locus.core.registry import ToolRegistry + from locus.tools._paths import file_tool_kwargs + from locus.tools._stubs import BaseTool + + tools = [] + for name in ASSISTANT_TOOLS: + if not ToolRegistry.contains(name): + continue + tool_cls = ToolRegistry.get(name) + if isinstance(tool_cls, BaseTool): + tools.append(tool_cls) + continue + if not (isinstance(tool_cls, type) and issubclass(tool_cls, BaseTool)): + continue + kwargs = file_tool_kwargs(name, config) + # An empty allow-list means "anywhere"; keep Agent mode in the home + # folder unless the user configured something narrower or wider. + if name == "file_read" and not kwargs.get("allowed_dirs"): + kwargs["allowed_dirs"] = [str(Path.home())] + tools.append(tool_cls(**kwargs)) + return tools + + +def _build_task_agent(state: Any) -> Any: + import inspect + + import locus.agents # noqa: F401 # trigger registration + from locus.core.registry import AgentRegistry + from locus.security.runtime import agent_security_kwargs, wire_agent_security + + config = getattr(state, "config", None) + bus = getattr(state, "bus", None) + capability_policy = getattr(state, "capability_policy", None) + rate_limiter = getattr(state, "rate_limiter", None) + + agent_cls = AgentRegistry.get("orchestrator") + kwargs: dict[str, Any] = { + "bus": bus, + "tools": _assistant_tools(config), + "agent_id": ASSISTANT_AGENT_ID, + } + if config is not None: + kwargs["max_turns"] = config.agent.max_turns + if "prompt_builder" in inspect.signature(agent_cls.__init__).parameters: + from locus.prompt.builder import SystemPromptBuilder + + kwargs["prompt_builder"] = SystemPromptBuilder( + agent_template=config.agent.default_system_prompt or "", + memory_files_config=config.memory_files, + system_prompt_config=config.system_prompt, + ) + kwargs.update( + agent_security_kwargs( + agent_cls, + capability_policy=capability_policy, + rate_limiter=rate_limiter, + agent_id=ASSISTANT_AGENT_ID, + ) + ) + + agent = agent_cls(state.engine, state.model, **kwargs) + wire_agent_security( + agent, + bus=bus, + capability_policy=capability_policy, + rate_limiter=rate_limiter, + agent_id=ASSISTANT_AGENT_ID, + ) + return agent + + +def agent_event_ids(agent: Any) -> set[str]: + """Ids that *agent*'s turn and tool events carry in their ``agent`` field.""" + ids = { + getattr(agent, "agent_id", None), + getattr(agent, "_runtime_agent_id", None), + getattr(getattr(agent, "_executor", None), "_agent_id", None), + } + return {i for i in ids if isinstance(i, str) and i} diff --git a/tests/server/test_agent_mode.py b/tests/server/test_agent_mode.py new file mode 100644 index 0000000..90959c9 --- /dev/null +++ b/tests/server/test_agent_mode.py @@ -0,0 +1,249 @@ +"""Agent mode (``mode`` on /v1/chat/completions) and live agent events.""" + +from __future__ import annotations + +import json +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import MagicMock + +import pytest + +fastapi = pytest.importorskip("fastapi") + +from fastapi.testclient import TestClient # noqa: E402 + +from locus.agents._stubs import AgentResult # noqa: E402 +from locus.core.events import EventBus, EventType # noqa: E402 +from locus.server import task_agent # noqa: E402 +from locus.server.app import create_app # noqa: E402 + + +def _engine(seen: list): + engine = MagicMock() + engine.engine_id = "mock" + engine.health.return_value = True + engine.list_models.return_value = ["test-model"] + + async def stream(messages, *, model, temperature=0.7, max_tokens=1024, **kwargs): + seen.append(list(messages)) + for token in ["Hi", "!"]: + yield token + + engine.stream = stream + return engine + + +def _config(): + from locus.core.config import JarvisConfig + + cfg = JarvisConfig() + cfg.analytics.enabled = False + cfg.traces.enabled = False + cfg.security.enabled = False + cfg.agent.context_from_memory = False + return cfg + + +class FakeAgent: + """A tool agent that publishes its own events and one from another agent.""" + + agent_id = "fake" + + def __init__(self, bus=None, tools=True): + self._bus = bus + self._tools = [object()] if tools else [] + self._model = "test-model" + self.inputs: list[str] = [] + + def run(self, input, context=None): + self.inputs.append(input) + if self._bus is not None: + self._bus.publish( + EventType.TOOL_CALL_START, + {"tool": "file_read", "arguments": {"path": "a.txt"}, "agent": "fake"}, + ) + self._bus.publish( + EventType.TOOL_CALL_START, + {"tool": "shell_exec", "arguments": {}, "agent": "someone-else"}, + ) + self._bus.publish( + EventType.TOOL_CALL_END, + { + "tool": "file_read", + "success": True, + "latency": 0.01, + "result": "hello", + "agent": "fake", + }, + ) + return AgentResult(content="Done.") + + +def _post(client, **extra): + body = { + "model": "test-model", + "messages": [{"role": "user", "content": "read a.txt"}], + "stream": True, + **extra, + } + return client.post("/v1/chat/completions", json=body) + + +def _events(resp): + out, name = [], None + for line in resp.text.split("\n"): + if line.startswith("event: "): + name = line[7:].strip() + elif line.startswith("data: "): + if name: + out.append((name, json.loads(line[6:]))) + name = None + return out + + +def _content(resp): + text = "" + for line in resp.text.split("\n"): + if line.startswith("data: ") and line[6:] != "[DONE]": + data = json.loads(line[6:]) + for choice in data.get("choices", []): + text += choice.get("delta", {}).get("content") or "" + return text + + +def test_chat_mode_skips_the_configured_tool_agent(): + seen: list = [] + agent = FakeAgent() + app = create_app(_engine(seen), "test-model", agent=agent, config=_config()) + + resp = _post(TestClient(app), mode="chat") + + assert resp.status_code == 200 + assert agent.inputs == [] + assert seen, "chat mode should stream from the engine" + assert _content(resp) == "Hi!" + + +def test_agent_mode_reuses_a_configured_tool_agent(): + bus = EventBus() + agent = FakeAgent(bus=bus) + app = create_app(_engine([]), "test-model", agent=agent, bus=bus, config=_config()) + + resp = _post(TestClient(app), mode="agent") + + assert agent.inputs == ["read a.txt"] + assert _content(resp) == "Done." + + +def test_agent_mode_builds_an_agent_when_the_configured_one_has_none(monkeypatch): + seen: list = [] + bus = EventBus() + built = FakeAgent(bus=bus) + monkeypatch.setattr(task_agent, "_build_task_agent", lambda state: built) + simple = FakeAgent(tools=False) + app = create_app( + _engine(seen), "test-model", agent=simple, bus=bus, config=_config() + ) + client = TestClient(app) + + _post(client, mode="agent") + _post(client, mode="agent") + + assert built.inputs == ["read a.txt", "read a.txt"] + assert simple.inputs == [] + assert app.state.task_agent is built + # Without a mode the request still goes where it did before. + _post(client) + assert seen + + +def test_agent_mode_streams_only_its_own_tool_events(): + bus = EventBus() + agent = FakeAgent(bus=bus) + app = create_app(_engine([]), "test-model", agent=agent, bus=bus, config=_config()) + + events = _events(_post(TestClient(app), mode="agent")) + + starts = [data for name, data in events if name == "tool_call_start"] + ends = [data for name, data in events if name == "tool_call_end"] + assert [s["tool"] for s in starts] == ["file_read"] + assert json.loads(starts[0]["arguments"]) == {"path": "a.txt"} + assert [(e["tool"], e["success"], e["result"]) for e in ends] == [ + ("file_read", True, "hello") + ] + + +def test_agent_mode_with_web_search_answers_without_the_agent(monkeypatch): + from locus.server import web_context + + seen: list = [] + agent = FakeAgent() + + async def no_results(query): + return web_context.WebSearchResult(query=query, sources=[], error="offline") + + monkeypatch.setattr(web_context, "search_web", no_results) + app = create_app(_engine(seen), "test-model", agent=agent, config=_config()) + + _post(TestClient(app), mode="agent", web_search=True) + + assert agent.inputs == [] + assert seen + + +def test_agent_mode_falls_back_to_chat_when_the_agent_cannot_start(monkeypatch): + seen: list = [] + + def broken(state): + raise RuntimeError("no orchestrator") + + monkeypatch.setattr(task_agent, "_build_task_agent", broken) + app = create_app( + _engine(seen), "test-model", agent=FakeAgent(tools=False), config=_config() + ) + + resp = _post(TestClient(app), mode="agent") + + assert resp.status_code == 200 + assert _content(resp) == "Hi!" + + +def test_built_agent_uses_read_only_tools_kept_in_home(monkeypatch): + # The shared conftest clears registries between tests. + from locus.agents.orchestrator import OrchestratorAgent + from locus.core.registry import AgentRegistry, ToolRegistry + from locus.tools.calculator import CalculatorTool + from locus.tools.file_read import FileReadTool + from locus.tools.file_write import FileWriteTool + from locus.tools.think import ThinkTool + + AgentRegistry.register_value("orchestrator", OrchestratorAgent) + for name, cls in [ + ("think", ThinkTool), + ("calculator", CalculatorTool), + ("file_read", FileReadTool), + ("file_write", FileWriteTool), + ]: + ToolRegistry.register_value(name, cls) + + engine = MagicMock() + engine.engine_id = "mock" + state = SimpleNamespace( + engine=engine, + model="test-model", + config=_config(), + bus=EventBus(), + capability_policy=None, + rate_limiter=None, + ) + + agent = task_agent._build_task_agent(state) + + names = {tool.spec.name for tool in agent._tools} + assert names <= set(task_agent.ASSISTANT_TOOLS) + assert {"think", "calculator", "file_read"} <= names + assert not names & {"file_write", "shell_exec", "web_search", "http_request"} + file_read = next(t for t in agent._tools if t.spec.name == "file_read") + assert file_read._allowed_dirs == [Path.home().resolve()] + assert task_agent.ASSISTANT_AGENT_ID in task_agent.agent_event_ids(agent)