Skip to content

feat: window placement, Wayland, vision, voice approvals, wake word, signing and model uninstall - #37

Merged
aryanjsx merged 8 commits into
mainfrom
feat/agent-mode-polish
Nov 8, 2026
Merged

aryanjsx merged 8 commits into
mainfrom
feat/agent-mode-polish

Conversation

@aryanjsx

@aryanjsx aryanjsx commented Nov 8, 2026

Copy link
Copy Markdown
Owner

What does this PR do?

Fills the gaps left after 1.0.8, and adds model uninstall. Each part is its own commit.

Desktop control

  • Move, resize and snap windows: window_control gains move, resize and snap (halves, quarters, center). Positions are relative to the screen's usable area, windows stay on screen, and the minimum size is 200 px. Windows uses SetWindowPos, macOS uses System Events, and Linux uses wmctrl -e.
  • Window control on Wayland:
    • Sway uses swaymsg.
    • Hyprland uses hyprctl.
    • KDE Plasma uses kdotool.
    • On GNOME, which doesn't allow it, the tool says so. When the needed tool is missing, it names it.
  • Screenshots vision models can see: screenshot takes look=true and returns a JPEG of at most 1280 px. After each round of tools, the orchestrator adds tool images as one user message. Agent and developer mode send them only when the model has Ollama's vision capability.

Models

  • Agent model recommendation: GET /v1/agent-model suggests the largest installed model of 7B or more that fits in memory. Otherwise it suggests qwen3.5:9b from 12 GB, or qwen3.5:4b from 8 GB with a warning. With Agent mode on and a model under 7B, the chat input offers Use or Download and use.
  • Uninstall from Settings: Settings > Models lists installed models with their sizes and an In use badge. Uninstall asks inline first, reports the space freed, and switches chat to another model when needed. The model picker's bin is now visible and needs a second click to confirm.
  • Server routes: GET /v1/models/installed is new. DELETE /v1/models/{name} now returns freed_bytes and was_default, gives a 404 for a model that isn't installed, and clears the cached vision capability. Its size lookup is best-effort.

Voice

  • Approvals by voice: in a hands-free turn, Locus reads the step aloud ("Locus wants to set the volume to 30%. Should I go ahead? Say yes or no.") and listens for the answer. An unclear answer falls back to the buttons. Commands and code are read out but always need a click.
  • "Hey Locus" wake word: opt-in, under Settings > Speech. A segmenter finds short bursts of speech, and only bursts of 0.3–3.5 s are transcribed, by the local faster-whisper backend. It refuses to run with cloud speech backends and pauses while Locus is busy. "Hey Locus, mute the sound" sends the command directly.

Release

  • Windows code-signing: desktop.yml signs Windows installers with either a .pfx (WINDOWS_CERTIFICATE and WINDOWS_CERTIFICATE_PASSWORD) or Azure Trusted Signing (AZURE_*). With neither, it builds unsigned and warns on tags, as before. macOS signing was already wired to the APPLE_* secrets. docs/development/release-checklist.md lists every secret.

Things a reviewer should know

  • Signing: this needs a certificate or a Trusted Signing account, and repository secrets. It changes nothing until they're added.
  • Wake word: while it's on, the microphone stays open. Audio is transcribed only on this computer.
  • Wayland backends: tested with the compositor commands faked only; there's no Sway, Hyprland or KDE machine here.

How was this tested?

  • New Python tests:
    • window placement maths and per-OS commands, and the Wayland backends
    • screenshot look=true
    • tool images reaching vision models only
    • the model recommendation and /v1/agent-model
    • tests/server/test_uninstall_models.py, against a fake Ollama
  • New frontend tests: voice yes/no parsing, the wake-word matcher and segmenter, the model-hint helper, and the uninstall helpers.
  • Frontend: all 181 vitest tests pass, and tsc -b is clean.
  • Full Python suites: run per folder: server, tools, agents, cli, core, speech and security. Every failure also fails on main, mostly because the locus_rust extension isn't installed locally. The exception is test_tool_timeout.py::test_repeated_timeouts_use_bounded_workers, which depends on test order and passes alone.
  • On a Windows 11 PC:
    • read the screen work area for snapping, without moving any window
    • the signing step's config rewrite, on a sample config (no certificate imported)
    • Ollama 0.40.15 installed with qwen3.5:9b, which reports tools and vision capabilities
  • Not done: the end-to-end Agent mode test with the real model was stopped before it finished, and the wake word and voice approvals haven't been tried with a real microphone.

Checklist

  • Tests pass (uv run pytest tests/ -v): new and changed suites pass; remaining failures also fail on main
  • Linter passes (ruff check on changed files)
  • Formatter passes on changed files
  • New/changed public API has docstrings
  • Follows registry pattern (if adding new component)
  • Documentation updated (if applicable)

window_control gains move (x, y), resize (width, height) and snap
(left, right, top, bottom, the four corners, or center).

Coordinates are relative to the usable area of the window's screen
(without the taskbar or menu bar), and placements stay inside it with a
200 px minimum size, so a window can't end up off screen.

- Windows uses SetWindowPos with the monitor's work area.
- macOS uses System Events, below the menu bar.
- Linux uses wmctrl -e with the desktop's work area.
window_control on Linux used wmctrl, which only works on X11. On a
Wayland session it now talks to the compositor:

- Sway uses swaymsg (minimize moves to the scratchpad).
- Hyprland uses hyprctl (minimize uses a special workspace, since
  Hyprland has none).
- KDE Plasma uses kdotool.

Move, resize and snap use the focused screen's area. GNOME doesn't let
apps control other windows, so the tool says so, and it names the tool
to install when one is missing.
The agent could take a screenshot but only got its file path back. Now:

- screenshot takes look=true and also returns the screen as a base64
  JPEG of at most 1280 px a side (Pillow is already in the server and
  desktop extras).
- After each round of tools, the orchestrator shows the model any
  images the tools returned, as one user message, since tool messages
  can't carry images in most chat APIs.
- Agent and developer mode tell the agent whether the model can see
  images (Ollama's vision capability), so a text-only model gets only
  the text.
The desktop app sizes its default model by memory, which gives
qwen3.5:4b on 16 GB machines, and small models often miss or garble
tool calls. Agent mode now recommends a better model:

- GET /v1/agent-model picks the largest installed model of 7B or more
  that fits in memory. Otherwise it suggests qwen3.5:9b from 12 GB
  (qwen3.5:4b from 8 GB, with a warning), and says when the computer
  has too little memory.
- With Agent mode on and a model under 7B, the chat input shows the
  suggestion with one click: Use it if it's installed, or Download and
  use it.
- Cloud and custom models with no size in their name aren't flagged.
During a hands-free turn, an approval was the one step that still needed
the mouse. Now Locus also reads the step aloud ("Locus wants to set the
volume to 30%. Should I go ahead? Say yes or no.") and listens for the
answer:

- Yes, okay, go ahead or allow approves; no, don't, stop or cancel
  denies. An unclear answer ("yes no wait") falls back to the buttons.
- Commands and code (run-level steps) are read out but always need a
  click, so nothing risky runs on a misheard word.
- The on-screen buttons keep working the whole time.

Adds speakAndWait() to the TTS store, which resolves when playback ends.
Hands-free voice needed the shortcut to start. With Say "Hey Locus" on
(Settings > Speech, under Hands-free voice), saying "Hey Locus" starts
listening, and "Hey Locus, mute the sound" sends the command straight
away.

There's no separate wake-word model:

- A segmenter finds short bursts of speech against a noise floor that
  follows quiet moments.
- Only bursts of 0.3-3.5 s are transcribed, by the local faster-whisper
  backend, and checked for the name. Common mishearings like "Lokus" and
  "Hey Lotus" count; "the locus of control" doesn't.
- It refuses to run with a cloud speech backend, so audio never leaves
  the computer.
- It pauses while Locus is listening, answering or speaking.

It's off by default, and the microphone stays on while it's on.
macOS signing and notarization were already wired to the APPLE_*
secrets, but Windows installers were always unsigned, so SmartScreen
warned on first launch.

A new "Configure Windows signing" step supports either:

- a code-signing certificate (WINDOWS_CERTIFICATE, base64 .pfx, and
  WINDOWS_CERTIFICATE_PASSWORD), imported for the build and used by
  thumbprint with SHA-256 and a DigiCert timestamp, or
- Azure Trusted Signing (AZURE_* secrets and variables) through
  trusted-signing-cli as Tauri's signCommand.

It adds the setting to tauri.release.conf.json. With neither set, it
warns on release tags and builds unsigned, as before. The release
checklist documents every secret for Windows and macOS.
Models could only be removed with a hidden, unconfirmed trash icon in
the model picker. Settings listed them by calling Ollama's port
directly, which fails on a custom host.

- Settings > Models lists installed models with their sizes and an In
  use badge. Uninstall asks inline ("Uninstall and free 6.6 GB?")
  first, then reports the space freed. If chat was using the model, it
  switches to another installed chat model, or says to download one.
- The model picker's bin is visible now and needs a second click to
  confirm. Errors show instead of being swallowed.
- GET /v1/models/installed lists models through the server, with
  sizes, parameter counts and which one Locus started with.
- DELETE /v1/models/{name} now returns freed_bytes and was_default,
  gives a 404 for a model that isn't installed, and clears the cached
  vision capability. The size lookup is best-effort, so it can't block
  an uninstall.
@aryanjsx
aryanjsx merged commit f7fa625 into main Nov 8, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant