Skip to content

Attach files and images, search the web, and ask for the user's name - #25

Merged
aryanjsx merged 4 commits into
mainfrom
feat/attachments-web-search
Nov 1, 2026
Merged

aryanjsx merged 4 commits into
mainfrom
feat/attachments-web-search

Conversation

@aryanjsx

@aryanjsx aryanjsx commented Nov 1, 2026

Copy link
Copy Markdown
Owner

Summary

Three chat features:

  1. Attach files, documents and images to a message.
  2. An opt-in Web search toggle that answers with cited sources.
  3. A "What should Locus call you?" pop-up that replaces the leftover "Win a Mac mini" giveaway, so the assistant addresses the user by nickname.

Attachments

  • Server: POST /v1/attachments (src/locus/server/attachments.py) reads uploaded files and stores nothing:
    • Documents: text from PDF, Word, Excel, PowerPoint, OpenDocument, EPUB, HTML, RTF and notebooks, and from anything that decodes as text (Markdown, CSV, JSON, YAML, logs, code).
    • Images: converted to a resized JPEG.
    • Other binary files: described by name and type. Old .doc/.xls/.ppt files get a "save as a newer format" note.
  • Chat messages: they carry attachments (documents) and images. Documents are folded into the prompt within about 40K characters, newest message first, so they fit the 16K-token default context.
  • Images: forwarded to vision models. When Ollama reports that a model has no vision capability, the images are replaced with a note so the model can suggest a vision model, instead of the request failing. Images are also sent correctly on the direct Ollama and OpenAI paths, which used to drop them.
  • Chat box: an attach button, drag and drop, and paste. Sent messages show the files. File contents are kept in IndexedDB rather than localStorage (about 5 MB) and are deleted with the conversation.
  • Dependencies: pdfplumber and pillow are added to the desktop and server extras. Both were already in uv.lock, so no versions change.

Web search

  • Toggle: a Web search toggle next to Deep Research, off by default and remembered. The two are mutually exclusive.
  • Server: with it on, the server searches for the latest user message (src/locus/server/web_context.py, using the existing WebSearchTool), reads the top three pages, and adds them as context that the model cites as [1], [2]. A message containing a URL reads that page instead.
  • Citations: a web_search SSE event sends the sources before the answer, so the existing citation renderer links [n]. A "Searched the web for …" list appears under the answer. The search context is marked so it doesn't suppress the server's identity prompt.
  • Direct path: requests with web search or attachments go straight to the engine, because agents only see the latest message's text.

Name instead of the giveaway

  • Pop-up: "What should Locus call you?", with optional first name, last name and nickname. It shows once after setup. The side panel's "Leaderboard" section becomes "You", showing what Locus calls you and reopening the pop-up.
  • Storage: GET/PUT /v1/profile (src/locus/server/profile_routes.py) writes a marked block in the persona's USER.md and leaves the rest of the file alone. USER.md is already part of every system prompt, and the block tells the assistant to address the user by their nickname, or their first name if no nickname is set. This applies to the next message, in the app and the CLI.
  • Removed: the giveaway dialog, its analytics prompt and their tests, and the giveaway wording in the Savings dashboard, docs/leaderboard.md and docs/telemetry.md.

Testing

  • Backend: 48 new tests: every document format, image conversion, upload limits, the context budget, the vision fallback, and web search. The search tests check that sources arrive before the answer and that the identity prompt is kept. The profile tests cover saving and reading back, keeping other USER.md content, cleaning input, and the nickname reaching the system prompt. Server tests show no new failures against main.
  • Frontend: tsc --noEmit and all 118 frontend tests pass, including new tests for attachments and the profile.
  • In the browser, against a stand-in Ollama that reports what it receives:
    • Attachments and search: a Markdown file, a CSV and a PNG with web search on reached the model as 2 documents and 1 image, together with real web results. Five sources were listed, [1] was linked, and the files survived a reload.
    • Profile: saving the pop-up wrote the block to USER.md. The next message told the model to call the user "Ary", and the side panel showed "Locus calls you Ary".
  • Not tested: answer quality with a real model.

Attachments
- New POST /v1/attachments reads uploaded files (nothing is stored):
  text from PDF, Office Open XML, OpenDocument, EPUB, HTML, RTF,
  notebooks and anything that decodes as text; images are converted to
  a resized JPEG. Other binary files are described by name and type.
- Chat messages carry `attachments` (documents) and `images`. The server
  folds documents into the prompt within a size budget that gives the
  newest message priority, and forwards images to vision models. For an
  Ollama model without the vision capability, images are replaced with
  a note so the model can suggest a vision model.
- The chat box gets an attach button, drag and drop, and paste. Sent
  messages show the files. Their content is kept in IndexedDB rather
  than localStorage, and deleted with the conversation.

Web search
- New Web search toggle (off by default, remembered). With it on, the
  server searches for the latest message, reads the top pages, and adds
  them as context the model cites as [1], [2]. A `web_search` SSE event
  sends the sources first so the answer's citations link to them.
- Requests with web search or attachments go straight to the engine,
  since agents only see the latest message's text.

Also send images in the right shape on the direct Ollama and OpenAI
paths, and add pdfplumber and Pillow to the desktop and server extras.
Replace the first-launch "Win a Mac mini" dialog, left over from
upstream, with "What should Locus call you?": first name, last name and
nickname, all optional.

- New GET/PUT /v1/profile stores the name in a marked block in the
  persona's USER.md, leaving the rest of the file alone. The system
  prompt already includes USER.md, and the block tells the assistant to
  address the user by their nickname (or first name), so it applies to
  the next message, in the app and the CLI.
- The dialog shows once after setup. The side panel's Leaderboard
  section becomes "You", showing what Locus calls you and reopening the
  dialog.
- Remove the giveaway dialog, its analytics prompt and their tests, and
  the giveaway wording in the Savings dashboard and docs.
pdfplumber is now in the server extra CI installs, so the test fed its
fake PDF to the real parser. Simulate the missing dependency so the test
checks the ImportError it's named for either way.
@aryanjsx
aryanjsx merged commit 7c68886 into main Nov 1, 2026
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant