Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
ceebba2
docs: design for autonomous supervisor, voice layer, and cross-repo A…
web3dev1337 Jul 25, 2026
d5b2bc5
feat(atlas): cross-repo knowledge map with per-audience sharing
web3dev1337 Jul 25, 2026
89edc70
feat: fleet supervisor + speech output + free-form voice routing
web3dev1337 Jul 25, 2026
79c7953
fix(atlas): CLI crashed with EPIPE when piped to head
web3dev1337 Jul 25, 2026
841fdc7
docs: record what shipped and what is left in the design doc
web3dev1337 Jul 25, 2026
47e7ed6
feat(supervisor): fix it yourself, interrupt last
web3dev1337 Jul 26, 2026
7707ac2
feat(atlas): git-backed registry so the map follows you between machines
web3dev1337 Jul 26, 2026
de95f0b
docs: research — Codex app-server, Hermes Agent, and the Discord rebuild
web3dev1337 Jul 26, 2026
fae4166
feat(discord): ambient watcher that tracks team work and publishes st…
web3dev1337 Jul 26, 2026
fe9221d
feat(codex): speak the app-server protocol instead of scraping terminals
web3dev1337 Jul 26, 2026
e394b71
feat(atlas): write-back so the map improves instead of rotting
web3dev1337 Jul 26, 2026
0cd18a9
feat(ui): JARVIS panel — Alt+J for the whole picture
web3dev1337 Jul 26, 2026
61fa05e
feat(voice): full-duplex realtime loop against Codex threads
web3dev1337 Jul 26, 2026
6847a06
docs: app-server bridge, realtime voice, atlas write-back, JARVIS panel
web3dev1337 Jul 26, 2026
a339779
docs: correct stale autonomy default, record what remains
web3dev1337 Jul 26, 2026
6555d9e
fix(codex): app-server client crash on error notif + broken respawn
web3dev1337 Jul 26, 2026
ee251f3
fix(shutdown): stop supervisor, discord-watch and app-server on exit
web3dev1337 Jul 26, 2026
9c2244a
fix(atlas): scrub real private repo names from public template + skill
web3dev1337 Jul 26, 2026
2460cfb
fix(supervisor): stop auto-approving writes to exec-on-next-op targets
web3dev1337 Jul 26, 2026
75cff6e
fix(atlas): keep unscored highlights null instead of forcing quality 1
web3dev1337 Jul 26, 2026
b2f1762
fix(voice): filter realtime socket events by thread id
web3dev1337 Jul 26, 2026
2527df6
fix(discord): merge override config, fix NaN defaults, guard negated …
web3dev1337 Jul 26, 2026
3174e59
fix(discord,atlas): correct backfill slice + atomic state/registry wr…
web3dev1337 Jul 26, 2026
2e98651
fix(ui): guard Alt+J while typing + surface panel action failures
web3dev1337 Jul 26, 2026
4cc3f65
fix(atlas cli): preserve '=' in flag values, reject value-less --quality
web3dev1337 Jul 26, 2026
3d7190f
fix(atlas): a local note must not make a subscribed repo re-shareable
web3dev1337 Jul 26, 2026
34f2882
docs: note the app-server thread linkage is still dormant
web3dev1337 Jul 26, 2026
f1449c4
fix(speech): prune the repeat-dedup map so it can't grow unbounded
web3dev1337 Jul 26, 2026
bab9037
fix: proposals atomic write, app-server buffer overflow, supervisor r…
web3dev1337 Jul 26, 2026
e658032
docs: design doc ladder matches the code, not an earlier draft
web3dev1337 Jul 26, 2026
6d657bd
docs: codebase map — correct autonomy default, complete test-file lis…
web3dev1337 Jul 26, 2026
fefe095
fix(atlas): sync reports the real commit/fetch failure, not a downstr…
web3dev1337 Jul 26, 2026
42b453c
fix(codex): stale-child race on quick restart + re-handshake on every…
web3dev1337 Jul 26, 2026
3906935
fix(codex): approvals with numeric ids are answerable over the API
web3dev1337 Jul 26, 2026
39661fa
fix(atlas): machine-local config must not sync with the registry
web3dev1337 Jul 26, 2026
239fb9e
fix(ui): clamp future timestamps so relative time never shows negativ…
web3dev1337 Jul 26, 2026
ee95384
fix(voice): make the live speech transcript actually readable
web3dev1337 Jul 26, 2026
e2723af
docs: research local full-duplex voice models + swappable-provider plan
web3dev1337 Jul 26, 2026
7dcfff2
feat(voice): swappable voice-model registry + local Piper TTS
web3dev1337 Jul 26, 2026
8e91051
docs: full voice-model catalogue + swappable-registry usage; codebase…
web3dev1337 Jul 26, 2026
6cecd4c
feat(voice): voice brain — Commander extension with fast fact answers
web3dev1337 Jul 26, 2026
5dc2e21
docs: record the voice brain in the codebase map
web3dev1337 Jul 26, 2026
ad209a2
feat(voice): speak the Commander's reply back — full two-way loop
web3dev1337 Jul 27, 2026
830c4cb
fix(voice): force JSON output from the local LLM so small models clas…
web3dev1337 Jul 27, 2026
c0cd953
docs: full session handoff — PR #1029 fixes + voice system, how to ru…
web3dev1337 Jul 27, 2026
22bba41
fix(voice): stop the local LLM jamming chit-chat into commands
web3dev1337 Jul 27, 2026
21638e2
fix(voice): order fact lanes before the greeting fallback
web3dev1337 Jul 27, 2026
3315f3a
fix(voice): stream the neural voice to the browser so it's actually a…
web3dev1337 Jul 27, 2026
35b7077
docs: handoff — WSL browser-audio streaming for the local voice
web3dev1337 Jul 27, 2026
5a59643
perf(voice): fact lane before the LLM + prewarm + auto-start Commander
web3dev1337 Jul 27, 2026
9ce8486
perf(voice): warm piper HTTP server -> ~0.2s synth instead of ~5s
web3dev1337 Jul 27, 2026
5fdc51f
docs: handoff — voice latency stack, warm piper, auto-start Commander…
web3dev1337 Jul 27, 2026
d3f6e9a
fix(voice): answer identity questions instantly, not via the Commander
web3dev1337 Jul 27, 2026
a66e2e5
feat(voice): Kokoro natural neural TTS via warm HTTP server
web3dev1337 Jul 27, 2026
10c66ea
fix(voice): never obey a negated command; ack dismissals instantly
web3dev1337 Jul 27, 2026
23ca056
perf(voice): resolve a pinned provider without probing every server
web3dev1337 Jul 27, 2026
3ccca0c
fix(voice): use the 3b model by default; skip the LLM for stray singl…
web3dev1337 Jul 27, 2026
79e65d9
fix(voice): rule-match 'open the <panel>' so the LLM can't misfile it
web3dev1337 Jul 27, 2026
0a457ca
fix(voice): answer natural fleet-status questions instantly
web3dev1337 Jul 27, 2026
593c66a
docs(voice): record natural-voice activation + routing hardening in h…
web3dev1337 Jul 27, 2026
dd820ed
fix(codex): contain stdio stream errors, real crash-loop backoff, und…
web3dev1337 Aug 5, 2026
856483a
fix(atlas): compile bundles from your own layers only — cloning a sha…
web3dev1337 Aug 5, 2026
50aa6ba
fix(speech): dead neural backends can no longer mute the voice silently
web3dev1337 Aug 5, 2026
e1ec3a0
docs: scrub remaining private repo names from files this PR adds to a…
web3dev1337 Aug 5, 2026
829fa7d
fix(voice): imperative 'stop/cancel <thing>' is a command, not a nega…
web3dev1337 Aug 5, 2026
3756f6c
fix(voice): negated destructive phrases can no longer execute; harden…
web3dev1337 Aug 5, 2026
6baec23
fix(supervisor): close permission-classifier bypasses; clear healed c…
web3dev1337 Aug 5, 2026
ff3b172
fix(discord): real guild permalinks, backward backfill paging, persis…
web3dev1337 Aug 5, 2026
30fec9d
docs: record the 2026-08-05 full-PR review pass in the handoff
web3dev1337 Aug 5, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions .repo-atlas.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
{
"id": "agent-workspace",
"name": "Agent Workspace",
"summary": "Multi-workspace orchestrator for running many CLI coding agents in parallel across repos and git worktrees. Express + Socket.IO backend, web client, Tauri desktop app.",
"kind": "tool",
"platforms": ["node"],
"languages": ["JavaScript"],
"dimension": "n/a",
"tags": ["orchestration", "agents", "worktrees"],
"status": "active",
"maturity": "production",
"visibility": "public",
"groups": [],
"highlights": [
{
"topic": "agent-workflow",
"quality": 5,
"paths": ["server/agentManager.js", "server/agentSpawnHelper.js", "config/custom-agents.example.json"],
"notes": "Agent-agnostic launch registry — new CLI agents are pure config (flags, model/effort syntax, init delay), no code changes."
},
{
"topic": "tooling",
"quality": 5,
"paths": ["server/repoAtlasService.js", "server/atlas/", "scripts/atlas.js"],
"notes": "The Repo Atlas itself: layered discovery/manifest/registry model with audience-scoped bundle compilation."
},
{
"topic": "testing",
"quality": 4,
"paths": ["tests/unit/", "jest.config.js"],
"notes": "700+ fast unit tests over services; Playwright e2e runs on a dedicated port via scripts/run-e2e-safe.js."
},
{
"topic": "ci",
"quality": 4,
"paths": ["scripts/release/", "scripts/tauri/run-tauri-build.js"],
"notes": "Cross-platform Tauri packaging with version-drift guards and bundle filename verification before release upload."
},
{
"topic": "security",
"quality": 4,
"paths": ["server/utils/shellSafety.js", "server/policyService.js", "server/networkSecurityPolicy.js"],
"notes": "Allowlist validators for anything interpolated into a shell command, plus role policy and loopback-by-default bind rules."
}
],
"avoid": [
{ "topic": "architecture", "reason": "server/index.js is a ~9k-line route god-file; follow server/routes/ and the service modules instead" }
],
"seeAlso": []
}
155 changes: 155 additions & 0 deletions CODEBASE_DOCUMENTATION.md

Large diffs are not rendered by default.

261 changes: 261 additions & 0 deletions PLANS/2026-07-26/AUTONOMOUS_ORCHESTRATOR_VOICE_AND_REPO_ATLAS.md

Large diffs are not rendered by default.

193 changes: 193 additions & 0 deletions PLANS/2026-07-26/RESEARCH_HERMES_CODEX_AND_DISCORD.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,193 @@
# Research: Hermes Agent, the Codex app-server, and the Discord rebuild (2026-07-26)

Three questions, one conclusion: the most valuable thing found here is not a product to adopt,
it is a **protocol we can already speak**.

---

## 1. The Codex finding (this is the important one)

`openai/codex` is **Apache 2.0 and open source**, and the CLI already installed on this machine
ships a component called `codex app-server`. From its own README:

> Similar to MCP, `codex app-server` supports bidirectional communication using JSON-RPC 2.0
> messages. Supported transports: stdio, websocket, unix socket.

This is the interface that powers the Codex VS Code extension and the Codex app. It is documented,
versioned (`v1`/`v2` protocol modules), and speakable by anything that can write JSON lines.

### What it exposes that we currently guess at

The supervisor today infers agent state by regex-scraping terminal output — "Do you want to
proceed" means a permission prompt, a cost line means a turn ended. The app-server emits these as
**structured events**:

| We currently scrape | app-server emits |
|---|---|
| "Do you want to proceed?" | `item/commandExecution/requestApproval` |
| Cost/summary line = done | `turn/completed`, `item/completed` |
| Busy/idle heuristics on buffer growth | `thread/status/changed` |
| Nothing — invisible to us | `thread/tokenUsage/updated` |
| Nothing — invisible to us | `turn/plan/updated`, `turn/diff/updated` |
| Guessed from banner text | `account/rateLimits/updated` |

Every one of those is a supervisor signal we are currently reconstructing unreliably from a byte
stream. Approval prompts in particular: instead of pattern-matching prose that changes between
releases, we would receive the actual command and answer it over the wire.

### And the voice pipeline is right there

```
thread/realtime/start thread/realtime/appendAudio thread/realtime/outputAudio/delta
thread/realtime/appendText thread/realtime/transcript/delta thread/realtime/transcript/done
thread/realtime/listVoices thread/realtime/sdp thread/realtime/stop
```

`sdp` means WebRTC. This is the full-duplex voice OpenAI shipped to the Codex desktop app on
2026-07-23 — the same thing described as "orchestrate multi-threaded coding jobs by voice" — and it
is addressable locally.

### Answering the three questions directly

1. **Is there an API?** Two. The **Codex SDK** (TypeScript and Python) embeds the agent in your own
app. The **app-server protocol** drives a local Codex the way the official app does. Codex CLI
can also run as an MCP server.
2. **Can we reverse-engineer it?** No need. It is Apache 2.0 with the protocol documented in-repo.
3. **Can we do broader/better?** Yes, and this is the actual opportunity. **The Codex app is
Codex-only, and macOS-only.** Ours is agent-agnostic and cross-platform. So:

> **Speak app-server for Codex sessions to get structured signals; keep PTY scraping as the
> universal fallback for Claude, Gemini, aider and anything else. One supervisor, best-available
> signal per agent.**

That is a strictly better position than either product: OpenAI cannot generalise to Claude, and we
would not be throwing away the agent-agnostic layer to get the fidelity.

### Cost note

The app-server drives the **local Codex CLI**, which bills the Codex subscription — the same
"drive the CLI, don't call the API" property the rest of this system relies on.

### Recommended next step

An adapter seam in the supervisor: `signalSource: 'app-server' | 'pty'` per session, resolved from
the agent registry. PTY stays the default and the fallback; nothing regresses if the app-server is
unavailable. Sized at roughly a day, and it upgrades every condition in the rule table at once.

---

## 2. Hermes Agent — worth knowing, not worth adopting

[Hermes Agent](https://hermes-agent.nousresearch.com/docs/) (Nous Research, MIT, launched 2026-02-25)
is model-agnostic and self-hostable on Linux, macOS, WSL2, Windows and Android/Termux. It runs as a
CLI, a desktop app, an **OpenAI-compatible API server**, and a **gateway across 20+ messaging
platforms** — Telegram, Discord, Slack, WhatsApp, Teams. It works with Nous Portal, OpenRouter,
OpenAI, Anthropic, Gemini, DeepSeek, Qwen, or any OpenAI-compatible endpoint including Ollama.

**Can Codex run it?** Not in the sense of "Hermes powered by your Codex subscription". Hermes wants
an OpenAI-compatible `/v1/chat/completions` endpoint; the Codex CLI is not one. You could put a
proxy in between, but then you are paying per token through whatever the proxy talks to, which
throws away the subscription-billing advantage that makes continuous autonomy affordable here.

**Is it still beneficial?** Two things about it are genuinely interesting, and neither requires
adopting it:

1. **The messaging gateway.** 20+ platforms with one integration is real engineering we would not
want to redo. If the Discord bridge ever needs to become a Slack/Telegram/WhatsApp bridge, look
here first — as a component, behind our own work model.
2. **The self-improving skill loop** (agent-curated `MEMORY.md`, skills it writes and then refines
during use). That is the same instinct as Atlas write-back: the system recording what it learned
so the next run starts smarter. Worth stealing as a pattern.

**What is not useful:** its agent runtime. We already have one, and ours knows what a worktree is,
what tier a task is, and which PR is waiting on evidence. Running a second runtime that knows none
of that adds a process without adding capability.

**Verdict: no.** Revisit only if multi-platform messaging becomes the requirement.

---

## 3. Discord: what is wrong and what replaces it

### What exists today

`server/discordIntegrationService.js` is a **file-drop queue**. An external bot repo writes
`~/.claude/discord-queue/pending-tasks.json`; the orchestrator ensures a Services workspace and
sends a processing prompt to a Claude terminal. It has real hardening (signed queue verification,
idempotency keys, an audit log) but the *shape* is wrong:

- **It only sees what was explicitly queued.** Someone has to address the bot. Ordinary conversation
— which is where the actual assignments happen — is invisible.
- **A restart loses whatever arrived while it was down.** There is no cursor and no backfill.
- **Hardcoded paths** to another repo's queue directory.
- **It runs on one laptop**, so "is the bot up?" is a question with a real answer.

### The team-coordination gap underneath it

The tooling is Discord + Trello + GitHub, and between them nothing answers:

- What is the **priority** of what I just asked someone to do?
- Is anyone **working on it**, right now?
- Is their **agent** running, or did they forget to prompt it?
- Did a ticket ever get **created**?
- Did the work **land**?

Every one of those is knowable — the orchestrator already knows session status, tier, branch, and PR
state — it is just never published anywhere the team can see.

### The replacement, in three parts

**a) Durable ingest, cursor-based.** Poll `GET /channels/{id}/messages?after={lastSeenId}` instead
of holding a gateway socket. This is the fix for "it couldn't pick up what it missed", and it is a
fix by *construction* rather than by retry logic: there is no such thing as a missed message when
your read position is persisted. A restart after three days is just a longer page-through. Polling a
team chat every 10 seconds is entirely adequate and removes a whole class of connection-state bugs.

**b) Ambient extraction.** Read every message, not just mentions. A cheap rule pass finds the
obvious cases (a mention plus an imperative, a question directed at someone, a link to a PR or
card). Only the ambiguous middle goes to a model, batched — same rules-first/LLM-on-event economics
as the supervisor. Output is a **work item**: who, what, priority, source message, permalink.

**c) Status publishing — the part that closes the loop.** When a session picks up a work item, the
orchestrator posts back to the thread. When the branch pushes, when the PR opens, when it merges.
Nobody types a status update; the status *is* the system's own knowledge, published. "Is their agent
working on it" stops being a question you have to ask.

### Where does it run, and on whose computer

Honest answer: **it does not need a VPS to start, and it should not start with one.**

- The **ingest is stateless given its cursor.** Whoever's orchestrator is designated the hub polls
and publishes. If that machine sleeps, nothing is lost — it catches up on wake. That is a very
different failure mode from a dropped gateway connection.
- **Everyone else's orchestrator only publishes its own status**, which needs no inbound
connectivity at all.
- The upgrade path, if and when the hub machine being asleep becomes annoying: move *only* the
ingest and cursor to a cheap always-on box. It is a poller with a JSON file. The agents stay on
the machines that have the code and the credentials — those never move, because that is where the
work is.

Deliberately **not** proposed: a shared server that runs agents. Auth doesn't travel, worktrees
don't travel, and the moment the brain is remote you are proxying every signal it needs back to it.

---

## 4. What this changes about the roadmap

1. **Discord ambient ingest + work items + status publishing** — replaces the file-drop queue.
2. **App-server adapter for Codex sessions** — upgrades every supervisor condition at once by
replacing scraped signals with structured ones. Highest ratio of capability to effort on this
list.
3. **Realtime voice via `thread/realtime/*`** — full-duplex voice for Codex threads, using the same
pipeline OpenAI shipped, without being locked to their app or to macOS.
4. **Atlas write-back** (agents propose highlights from work they just did) — the Hermes
self-improving-skills idea, applied to the map.

## Sources

- [openai/codex](https://github.com/openai/codex) — Apache 2.0; `codex-rs/app-server/README.md` documents the protocol
- [Codex SDK](https://developers.openai.com/codex/sdk) · [Codex with the Agents SDK](https://developers.openai.com/codex/guides/agents-sdk)
- [Introducing the Codex app](https://openai.com/index/introducing-the-codex-app/)
- [VentureBeat — GPT-Live full-duplex voice control comes to Codex](https://venturebeat.com/orchestration/agentic-coding-goes-hands-free-as-openai-brings-gpt-lives-full-duplex-voice-control-to-codex-and-chatgpt-on-the-desktop)
- [Hermes Agent docs](https://hermes-agent.nousresearch.com/docs/) · [AI providers](https://hermes-agent.nousresearch.com/docs/integrations/providers)
- [Hermes Agent vs OpenClaw comparison](https://contabo.com/blog/hermes-agent-vs-openclaw-paperclip-and-the-best-open-source-ai-agents-in-2026/)
Loading
Loading