feat: JARVIS — self-healing fleet supervisor, ambient Discord work tracking, git-backed Repo Atlas - #1029
Open
web3dev1337 wants to merge 69 commits into
Open
feat: JARVIS — self-healing fleet supervisor, ambient Discord work tracking, git-backed Repo Atlas#1029web3dev1337 wants to merge 69 commits into
web3dev1337 wants to merge 69 commits into
Conversation
…tlas Investigates three linked asks: a top-level orchestrator that watches the agent fleet, voice as the primary interface, and a cross-repo knowledge map with per-audience access scoping. Key findings: - Rules-in-a-loop / LLM-on-event keeps continuous supervision at ~0 tokens - Driving the agent CLI in a PTY bills the subscription, not the API, so autonomous Claude does not require API credits - OpenClaw and Hermes Agent are the wrong shape: neither models worktrees, tiers, queue or review state, which is the entire value here - Atlas quality must be per-topic, not per-repo: a rough prototype can still hold the best example of one thing
The problem: hundreds of repos, a fraction cloned locally, and no way for an agent to know that a scruffy prototype holds the best test harness you own. Grepping ~/GitHub finds only what happens to be on disk and cannot express quality. The Repo Atlas is CODEBASE_DOCUMENTATION.md one level up: - Layered entries — discovery (disk + gh repo list) < in-repo .repo-atlas.json < your registry override. Your opinion always wins. - Cloned-ness is irrelevant. A repo that only exists on GitHub is a first-class breadcrumb with a clone hint attached. - Quality is per-topic, not per-repo, so 'don't copy this, except exactly this one thing' is expressible — which is the truth about real codebases. - Sharing is subtractive: entries compile down into audience bundles. private never leaves the machine and overrides group membership; team needs a group match; public goes everywhere. Local paths are always stripped, and per-group redaction can list a repo while hiding its internals. - 'atlas digest' emits a terse map to paste into a prompt, so an agent knows where to look without spending tokens finding out. Surfaces: standalone CLI (no server needed), /api/atlas/* routes, agent skill. 42 new unit tests; 652 total green.
Three pieces of the same thing: an assistant that notices first, and that you can talk to. SUPERVISOR Nothing in the orchestrator was push-based. pagerService nudges on a fixed interval whether or not anything is wrong; schedulerService runs on a clock; processAdvisorService computes good advice but only when a human opens the panel. You had to notice a stuck agent yourself. The supervisor closes that loop without spending tokens to do it. Sensors and rules run every tick — PTY tail, session status, how long a buffer has been quiet, git ahead/dirty for quiet sessions only — and a model is never called in the loop, only on escalation. That is what makes it affordable to leave running permanently. Findings climb observe -> notify -> nudge -> act, capped by an autonomy level: - observe is the shipped default and has zero side effects, so the rules can be judged from a week of findings before being trusted with anything - no shipped condition reaches 'act' (asserted in tests) - act handlers are named functions, so a rule file cannot inject shell - auto-answering a permission prompt fails closed: any deny-pattern match, or no allow-pattern match at all, escalates to a human instead of guessing - per-finding cooldowns, and every action lands in supervisor-audit.jsonl Detects: permission prompts left hanging, usage limits, error loops, stalls, exited agents, unpushed and uncommitted work, idle capacity. SPEECH OUT Backends degrade: browser speech synthesis by default (nothing to install, so it works on a fresh clone), with piper/say/SAPI/espeak preferred when present. Text is sanitized to printable ASCII with no shell metacharacters before it can reach a command line, length-capped, and de-duplicated. FREE-FORM VOICE Previously an utterance matching no pattern was a dead end, which is what makes a voice interface feel like a remote control. Unmatched speech is now handed to the Commander agent, so the fallback for 'I didn't understand' is a full agent with the whole orchestrator API rather than an error beep. 709 unit tests green.
`atlas list | head` closes stdout early; that is a normal end for a CLI, not an unhandled error event.
The plan section read as intent; it is now the record — what landed where, how it was verified, why only two atlas entries were seeded, and the six things still open.
Correcting the shipped design. The previous ladder ended in 'tell the human',
which is backwards — being narrated at about things the system could have
fixed is worse than not having the system.
The ordering is now the whole point:
fix it myself -> hand it to the Commander -> (only then) interrupt
- Every shipped condition declares how to repair itself (asserted in tests).
A stall gets nudged, a dead agent gets relaunched, a usage limit parses its
own reset time and schedules a 'continue' — that last one used to be a
critical alert and is now an info-level non-event, because waiting is not a
problem you need to know about.
- A finding that has not exhausted its repair attempts is not even eligible to
reach a human. Only after N failed self-heals does it become escalatable.
- When rules run out of ideas, the problem goes to the Commander with a written
brief (label, branch, tier, ticket, output tail) before it goes to you. It is
a full agent with the whole API and only costs tokens when something is wrong.
- Urgency is weighted by task tier: the same stall scores 96 on T1 focus work
and 24 on T4 background work. Background work now structurally cannot pull
you out of flow.
- An interruption budget (2/hour, 15min apart, optional quiet hours) gates what
is left; anything refused goes to a batched digest instead of being dropped.
A high enough score overrides quiet hours and the hourly cap, but nothing
overrides the per-finding guard — no drumbeat about the same problem.
- Problems that heal are removed from the digest and forgotten. Solved problems
are never mentioned.
Default autonomy is now 'autopilot' (SUPERVISOR_AUTONOMY overrides). The safety
invariants are what make that defensible: named handlers only, fail-closed
permission approval that refuses credentials/force-push/merge, full audit.
Also fixes a Number(null) === 0 bug class in urgency config parsing that made
every unset numeric option silently zero.
752 unit tests green.
A map that only exists on one laptop is not a map you can rely on. The blocker was that syncing it collides with the sharing model — the master registry describes private repos, so it cannot just be pushed somewhere shared. The split that resolves it: - REGISTRY (portable judgement: what a repo is worth reading for, and who may see it) lives in a PRIVATE git repo you control. `atlas remote set` + `atlas sync` pulls, merges and pushes it. Multi-machine and backed up. - DISCOVERY (what this particular computer has cloned) stays local and is never synced — it would be wrong on every other machine. - BUNDLES (the subsets you share) are published into whichever repo that audience already has access to. GitHub permissions stay the enforcement. One file per repo under entries/. That detail is what makes it work: two machines curating different repos touch different files, so git merges them with no conflict at all — verified with a real remote and interleaved edits. Subscriptions close the loop the other way: `atlas subscribe` reads a bundle someone published, so their map shows up in your searches, attributed to them, at the lowest precedence — your own notes always win. And what was shared with you is never re-shared by you: foreign entries are excluded from compilation. Legacy single-file registries migrate automatically on first read. 762 unit tests green.
The Codex finding is the significant one. openai/codex is Apache 2.0 and the CLI already installed here ships `codex app-server`: JSON-RPC 2.0 over stdio/websocket/unix socket, the same interface that powers the VS Code extension and the Codex app. It emits, as structured events, everything the supervisor currently reconstructs by regex-scraping terminal output — requestApproval instead of matching 'Do you want to proceed', turn/completed instead of spotting a cost line, thread/status/changed instead of buffer-growth heuristics — plus token usage and rate limits we cannot see at all today. And thread/realtime/* is the full-duplex voice pipeline OpenAI shipped on 2026-07-23, WebRTC SDP included. Answers: yes there is an API (SDK + app-server); no reverse engineering needed; and the broader play is real — the Codex app is Codex-only and macOS-only, so speaking app-server for Codex while keeping PTY scraping as the universal fallback beats both products. Hermes Agent: model-agnostic and genuinely good, but it cannot run on the Codex subscription (it wants an OpenAI-compatible endpoint, which the CLI is not), and its runtime duplicates ours without knowing worktrees, tiers or queue state. Its messaging gateway and self-improving skill loop are worth stealing as patterns. Verdict: no. Discord: the current file-drop queue only sees what was explicitly queued and loses anything that arrived while it was down. Replacement is cursor-based polling (a missed message stops being possible rather than being retried), ambient extraction of work items, and status publishing so 'is their agent working on it' stops being a question anyone has to ask.
…atus
Replaces the file-drop queue. That design only saw what was explicitly queued,
lost anything that arrived while it was down, and hardcoded paths into another
repo — so the assignments that actually happen in conversation were invisible.
INGEST — cursor-based, not gateway-based
Polls `GET /channels/{id}/messages?after={lastSeenId}` with the position
persisted. A message missed while the process was down stops being a thing that
can happen, rather than a thing retry logic has to handle: a restart after three
days is just a longer page-through. Verified by killing and rebuilding the
service mid-test and confirming it resumes from the cursor without replaying.
EXTRACTION — read everything, track what matters
Rules over every message, LLM-free: a mention plus a request is an assignment
even though nobody filed a ticket; 'urgent'/'prod is down' is tier 1 and 'when
you get a chance' is tier 4; 'on it' claims the open item and 'done' closes it
rather than each creating more noise. Questions, bots and one-word replies are
ignored on purpose.
STATUS — the gap underneath the whole ask
The orchestrator already knows session status, tier, branch and PR state; it was
just never published anywhere the team could see. `linkSession()` binds a work
item to the session doing it, writes a task record with the right tier, and
announces it in-thread. 'Is their agent working on it' stops being a question
anyone has to ask, and /untracked is the list of what was asked for that nobody
has started — which until now existed only in scrollback.
Off by default; needs DISCORD_BOT_TOKEN and channel ids.
783 unit tests green.
web3dev1337
changed the title
The research said this was the highest-value next step, so here it is. `codex app-server` is Apache 2.0, ships in the CLI already installed, and is the JSON-RPC interface behind the Codex app and VS Code extension. Built and verified against the real thing, not against the docs: `initialize` handshake, `thread/list` (which returns `data`, not `threads` — the schema names imply otherwise), and live notification flow. What this replaces: scraped guess -> reported fact 'Do you want to proceed' regex -> ThreadActiveFlag waitingOnApproval cost line means the turn ended -> turn/completed with status + duration buffer-growth busy/idle heuristics -> thread/status/changed (invisible) -> systemError thread state (invisible) -> token usage, rate limits Two new conditions use the structured facts: `structured-approval` fires in seconds because a thread reporting waitingOnApproval is not something you need to wait out a quiet-time threshold to be confident about, and `thread-system-error` delegates a runtime failure rather than nudging it. Approvals can now be granted over the wire with the actual command in hand, instead of typing a keystroke at whatever prompt happens to be on screen. Degradation is the design: `getSignalForSession` returning null means 'no better information than the PTY', so Claude, Gemini and aider are unaffected and everything still works with the app-server off. Opt in with CODEX_APP_SERVER=true. Also lays the realtime groundwork — thread/realtime/* wired for full-duplex voice, transports websocket and webrtc. 803 unit tests green.
An atlas nobody updates becomes another stale doc. But agents cannot be given
write access either — left to write freely, every repo an agent touched would
end up rated 5/5 and the quality scores would stop meaning anything, which is
the entire value.
So: agents propose, you decide. A proposal carries the evidence for its claim
('40 tests added in tests/unit, all green'), which is what makes reviewing one
take two seconds instead of requiring you to go and look. Approving writes
through the exact same path manual curation uses, so an approved proposal is
indistinguishable from a note you wrote yourself and syncs identically.
Guards that matter: quality clamps to 1-5 so an over-eager agent cannot invent
a 9; topic aliases normalize so proposals do not fragment the vocabulary; a
second proposal for the same topic supersedes the first rather than stacking.
The skill now tells agents to propose at the end of substantial work, and tells
them plainly that they cannot write directly and should not try.
815 unit tests green.
The APIs existed but nothing rendered them, which meant the system could only be read by curling it. Leads with 'handled' on purpose: that number rising while the waiting list stays empty is what the system working looks like. A dashboard that only ever shows problems trains you to read it as a problem list, which is the habit this whole design is trying to break. Four sections: what still needs you (the digest, with urgency and how many times it recurred), what the team asked for that nobody started, Atlas proposals with one-click approve/reject, and Atlas topic search. Browser-verified rather than eyeballed: headless Chrome under Xvfb confirms the assets load, the CSS applies, all four sections render, and clicking Approve writes through the API into the registry — 1 proposal in, 0 remaining after the click, entry present in the map. Zero console errors, zero failed requests. Styling follows the house rules: rem throughout, 100dvh, safe-area inset, 44px touch targets, one mobile breakpoint, and white/bright-accent text on dark with no grey-on-grey anywhere. 815 unit tests green.
Closes the loop OpenAI shipped to their desktop app on 2026-07-23, without being locked to their app or to macOS. Browser speech recognition transcribes you, the text goes to thread/realtime/appendText, and the assistant's transcript deltas arrive over the socket and are spoken by SpeechOutput. Hands-free, nothing to install. Two details that matter: only *completed* assistant turns are spoken, because speaking every delta stutters and speaking your own words back is absurd (browser-verified — 'Build is green.' spoken, the partial delta ignored); and continuous recognition restarts itself, because browsers end the stream periodically whether or not you are done talking. Honest about maturity: the raw-audio path to appendAudio is wired but the exact PCM framing has not been confirmed against a live authenticated realtime session, so it is opt-in and the text path is the default rather than pretending otherwise. The text path is a complete working voice loop today. 815 unit tests green; browser-verified under Xvfb with zero console errors.
The Commander reference still described 'observe' as the shipped default after that changed to autopilot, and the design doc's open-items list was written before most of it shipped.
Two bugs in the app-server bridge, both fatal when CODEX_APP_SERVER=true: 1. A protocol notification with method 'error' (a real, expected type — systemError threads) and any child-process error event were re-emitted as the EventEmitter-reserved 'error' event with no listener, which Node throws for. That surfaced as an uncaughtException and shut the whole orchestrator down, killing every session in every workspace over one Codex thread's problem. Add a default 'error' listener and stop re-emitting the reserved name (the 'error' notification is still delivered via 'notification'). 2. start() nulled this.starting inside the Promise executor, which runs synchronously and is immediately clobbered by the outer assignment — so this.starting held a stale resolved promise forever. Every later start() short-circuited on it, meaning auto-restart after a crash (and manual restart) never spawned again. Clear the marker after the promise settles. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
shutdown() only cleaned up sessions/io/http. The app-server child process (codex app-server) was never sent SIGTERM, so every nodemon reload or restart with CODEX_APP_SERVER=true leaked one orphaned process, invisible to a 'ps aux | grep node'. Stop all three new services first. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
config/repo-atlas.example.json and skills/public/repo-atlas/SKILL.md shipped with real PRIVATE repo names (zoo-game, box2d-luau, drain-the-lake, hyfire2, hytopia-client-tracker) plus a real local path exposing the OS username, all committed to this PUBLIC repo. The design doc's own goal for these artifacts is 'ships with zero personal data; an example manifest only'. Replace with clearly fictional placeholders (acme-tycoon, physics-kit, puzzle-proto, ~/GitHub/...). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Edit/Write/Update allow pattern had no path restriction, so with the shipped default autonomy 'autopilot' the supervisor would auto-approve a write to .git/hooks, package.json (scripts), .github/workflows, a shell rc, Makefile, ~/.npmrc or /etc — each of which executes on the next already-allowlisted 'npm run build'/'git commit', i.e. unattended RCE. Add path-based deny patterns so those fail closed to a human while ordinary source edits stay auto-approved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
qualityScore() ran Number(value) before the finite check, and Number(null) is 0, which clamped to 1. An intentionally-unscored highlight (quality: null, which atlasProposals explicitly supports for 'no opinion yet') therefore became the worst score once approved — inventing a cautionary rating the design doc's own 'a fabricated score is worse than a blank field' principle forbids. Guard the null-ish cases; a real out-of-range 0 still floors to 1 as before. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
app-server-transcript and app-server-realtime are broadcast to every client (io.emit), but both handlers acted on all of them. Two tabs on different Codex threads would hear each other's assistant replies spoken aloud, and one thread closing would stop another tab's listening loop. Ignore events whose threadId isn't this client's, matching the server-side getTranscripts() filter. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…signals
Three bugs in the ambient watcher's config + extraction:
- loadConfig picked the override file wholesale, so the documented minimal
override ({enabled, channels}) silently emptied every priority/kind/claim/
done/drop pattern table. Layer it on top of the defaults instead.
- Number(x) ?? default leaves NaN when the field is absent (?? only catches
null/undefined), so a missing backfillMessages/minLength became NaN. Use a
finite-check helper; an explicit backfillMessages:0 is still preserved.
- 'not done yet' / 'isn't fixed' matched the done patterns and closed an open
item. Guard complete/claim with a negation check (drop keeps its intentional
negatives like 'not needed').
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ites - fetchMessagesAfter kept the OLDEST N of a first-sight backfill (slice(0,N)) after fetching newest-first, so it skipped exactly the most recent messages a backfill exists to catch. Keep the tail (most recent N) when there's no cursor; forward paging still keeps the oldest N after the cursor. Also point the returned cursor at the last kept message so a >cap page can't skip the overflow. - atlasStore.writeJson and DiscordWatchService.saveState wrote straight to the final path; a crash mid-write corrupts the file. The atlas registry is git-synced, so a half-write would propagate everywhere. Write-then-rename. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Alt+J fired even inside inputs/textareas (including the panel's own atlas search box) and on Ctrl+Alt/AltGr layouts, swallowing the keystroke. Add the same typing/modifier guard every other Alt-shortcut in the app uses. - The catch-up / approve / reject buttons awaited network calls with no try/catch, so a failed request became an unhandled rejection and the button looked dead. Catch and show a dismissable error banner (white on dark red). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- parseArgs split flag tokens on every '=', truncating any value containing one (--notes="a = b" became "a "). Split on the first '=' only. - A forgotten --quality value parsed as boolean true and Number(true) silently became quality 1 (the worst score). Validate: a value-less or non-numeric --quality now errors instead of inventing a rating. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An entry was marked foreign (excluded from compiled bundles) only when it was subscription-ONLY. Adding any local registry note to a repo a teammate shared added a registry layer, cleared the foreign flag, and compile() then re-published the whole entry — including summary/highlights inherited from the teammate's bundle — defeating the 'subscriber provably unable to re-share' guarantee. Foreignness now depends on whether you actually have the repo locally (discovery), not on whether you annotated it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The bridge works against a real codex app-server, but no session-to-thread-id link exists yet, so the supervisor runs on PTY signals in practice. State that plainly rather than implying structured signals are live.
lastSpokenAt kept one entry per distinct spoken string forever. Over a long-lived server that is a slow leak. Drop entries past the repeat window (they can never match isRepeat again) each time a new one is recorded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…obustness Second review pass, lower-severity items confirmed by the scout reports: - atlasProposals.save wrote the whole proposal queue non-atomically (same class as the registry fix, bigger blast radius). Route through store.writeJson. - app-server consume() dropped the ENTIRE buffer on overflow, discarding any complete frames queued ahead of the oversized line. Process complete lines first, then drop only the oversized incomplete tail. - supervisor nudge conditions (stalled, unpushed-work, uncommitted-work) relied on array ordering to avoid typing into an exited agent's bare shell. Require agentPresent:true so the invariant holds regardless of operator reordering. - supervisorSignals tier used Number(x)||null (the 0-becomes-null gotcha this repo's CLAUDE.md calls out). Use a finite check. - voice-control showed a literal 'null' status when speech was forwarded to the Commander (no command name). Show 'Sent to Commander' instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… map Documents every option found (GPT-Live closed SOTA reference, PersonaPlex/Moshi/ X-Talk duplex, Qwen omni, Parakeet/faster-whisper/Moonshine STT, Kokoro/Chatterbox/ Piper TTS + more) and the shipped add/swap-a-model workflow. Records the new voiceProviderService/routes/config in CODEBASE_DOCUMENTATION.
Voice is no longer a fixed phrasebook. An utterance routes through three lanes,
fastest first:
1. COMMAND — a semantic command in the registry, run instantly (kept).
2. FACT — a question answerable from live orchestrator state (sessions,
supervisor briefing, queue, discord, workspace) answered straight from a
context snapshot: an API shortcut, no LLM turn, spoken back in ~ms.
3. AGENT — anything else handed to the Commander (full API, can do anything),
acknowledged aloud.
Every lane speaks. An action phrasing ('open the queue') is never hijacked by
the fact lane. Built on the existing commanderContextService snapshot + command
registry, so the voice has the same visibility the Commander does.
Verified live: 'how many agents working', 'what needs me', 'what workspace',
'what can you do' answered instantly from real state and spoken via local Piper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The agent lane now completes the conversation. On an open-ended request the brain
acks immediately ('On it.'), then in the BACKGROUND watches the Commander's PTY
buffer until its output settles, extracts the assistant's actual prose out of the
Claude Code TUI (strips ANSI, box-drawing, spinners, chrome, the echoed request),
and speaks it aloud. The request returns instantly; the answer arrives when the
agent is done — a real back-and-forth, not a dead-end ack.
extractAssistantReply is a pure, tested function (noisy-TUI-in, clean-prose-out,
null when there's no prose so it never speaks garbage).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sify reliably
parseWithOllama sent a free-form prompt; llama3.2:1b rambled prose and every
fuzzy phrasing fell through to 'no match'. Add Ollama's format:'json' (constrains
generation to the {"command":...} grammar at the API level) and widen the
timeout. Verified with llama3.2:3b: 'pull up the queue for me' -> open-queue,
'open the settings please' -> open-settings, etc.
Live testing exposed 'hello can you hear me' -> switch-workspace and 'thanks
that is cool' -> open-project-chats: a small model forced to emit JSON picks a
command for everything. Two guards:
- Prompt: explicit 'return {command:null} for greetings/questions/small talk',
with examples.
- Grounding: reject a classified command unless the utterance actually shares a
keyword with it (isGrounded), so a hallucinated command flows to the brain's
greeting/fact/agent lanes instead of firing.
Plus a greeting/presence handler in the brain ('are you there' -> 'Yes, I'm
here...'). Verified: chit-chat -> spoken reply, real commands still fire.
Also fixed the piper wrapper (dangling venv symlink -> python -m piper), so the
local neural voice actually plays instead of falling back to the robotic browser
voice.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'hey what needs me' was swallowed by the greeting matcher. Greetings now run last, so a real question opening with 'hey' hits its specific lane first.
…udible on WSL
Root cause of 'I don't hear anything': server-side PulseAudio (WSLg) doesn't
reach the user's Windows speakers, but browser audio does (the browser TTS was
audible). So when a browser is connected, synthesize with piper server-side and
STREAM the WAV over the socket ('speech-audio'); the client plays it (Audio
element, high-priority clips interrupt). Falls back to paplay only when nothing
is listening in a browser. Reads the model's real sample rate from its .json.
Verified: valid 126KB RIFF/WAVE emitted; 2 clients receive it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Questions ('how many agents working') were paying the multi-second LLM
command-classifier BEFORE reaching the instant fact lane — 8034ms observed,
worst case an 8s cold-model timeout. Answer facts from the snapshot FIRST;
measured 15-24ms now. The LLM classifier only runs for genuine command
phrasings.
- Prewarm the Ollama model on boot + keep_alive 30m so the first real command
doesn't eat the cold load; warm classification ~1.7s.
- sendToCommander auto-starts the Commander if none is running (launch queue
buffers the request through boot) instead of dead-ending on 'no Commander'.
- Log each utterance: heard / route / command / reply / ms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The felt delay was piper cold-starting (python -m piper reloads the model every call, ~5s). Run piper.http_server once (model stays loaded) and POST /synthesize to it (~0.2s), streaming the WAV to the browser. Falls back to spawning piper if the server is down. PIPER_HTTP_URL (default 127.0.0.1:5959). End-to-end synth measured at 242ms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'what is your name' was routing to the Commander agent (8s -> 'On it.'). Add an identity handler to the fact lane: 'who/what are you', 'your name' -> instant 'I'm JARVIS...' reply.
Adds a natural-sounding local voice (Kokoro) as the highest-quality TTS provider, served from a warm HTTP server (~2s CPU synth, faster on GPU) and streamed to the browser like piper so it is reliably audible on WSL. Registry health now checks the kokoro server, and speakLocally routes kokoro through the browser when a client is connected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two routing bugs found in stress testing: - "don't open the queue" was classified by the LLM into a queue command — doing the opposite of what was said. parseCommand now short-circuits a leading negation and hands it to the Commander (which understands "don't") instead of the classifier. - "never mind" / "actually never mind" fell through to the Commander because the action/negation guard swallowed the leading "never". The dismissal ack now runs before that guard and tolerates a leading filler word, so it answers instantly with "Okay, forget it." Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
resolveActive now health-checks only the pinned provider first and returns it if available, instead of HTTP-probing every model server just to confirm the one already chosen. Falls back to auto only when the pin is gone/unavailable, so a broken pin still never mutes voice. Tests made hermetic (fixed provider set) so resolution assertions don't depend on which real model servers happen to be up on the test machine. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e words
- The classifier was silently running on llama3.2:1b even when the tuned
3b was installed: the model-preference check was satisfied by ANY
llama3.2 tag, so it never upgraded off the 1b that rambles and misfiles
chit-chat as commands. Default to 3b and pick the best installed model
(qwen/3b before 1b/phi), degrading gracefully.
- A single mis-heard word ("uh") matched no rule and no fact but still
paid a ~900ms LLM round-trip only to fail. Short-circuit it so it falls
through instantly to "didn't catch that".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Live testing showed "open the queue" being classified by the 3b model as
queue-select-by-pr-ref — the wrong command within the queue family. The
auto-parser only matched the exact command name ("open queue"), so the
natural "open THE queue" / "pull up the queue" / "show me the queue"
phrasings fell through to the fuzzy LLM. Add deterministic rules for the
open-queue / open-tasks / open-advice / open-settings panels; each
requires its object noun so it never shadows the specific queue rules
("open blockers", "triage queue", "open next review").
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
"how's the fleet doing" fell through to the Commander (a ~1.2s LLM round-trip) instead of the instant fact lane, because the regex only matched "how many agents" / "fleet status". Broaden it to catch "how's the fleet", "how are the agents doing", etc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…andoff Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…elivered approvals stay pending - stdio pipes had no error listeners: an async EPIPE from a half-dead child (or any other stream error code) threw uncaught — only the literal EPIPE happened to be swallowed by the global handler; anything else took the whole orchestrator down. A stream error now retires that child so the normal exit/restart path takes over. - restartAttempts was zeroed at every spawn, so a binary that spawns fine but dies instantly read attempt 0 every cycle and was hammered at the 1s floor forever. Only sustained uptime (30s) resets the ladder now. - answerApproval deleted the pending entry even when the wire write failed (app-server mid-restart), making the approval vanish from the UI while the codex thread stayed blocked. An undelivered answer now keeps the approval pending and retryable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…red repo no longer republishes the sharer's judgement - compile() filtered on the `foreign` flag, which clears the moment discovery also knows a subscribed repo (you cloned it / gh lists it). From then on the teammate's highlights/summary rode into YOUR bundles, unattributed, to audiences they never authorized. Bundles now merge from discovery/manifest/registry only (getOwnEntries) — subscription content structurally cannot be re-shared, while a repo you genuinely have stays shareable with just your own fields. - writeManifest only knew the master/ worktree convention; for main/ layouts `atlas init` wrote .repo-atlas.json into the non-repo parent dir where it could never be committed or synced. - atlas CLI: a value-less --topic parsed as boolean true, slipped past the usage check, and was recorded as the literal topic "true" (--quality already had this guard; note/avoid/propose now do too). - qualityFloor treats bare booleans as "no floor" like the schema does, instead of coercing false to a floor of 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- kokoro reported 'available' unconditionally (its HTTP URL has a default, so Boolean(url) was always true). A dead kokoro was selectable, synth failures were swallowed with no fallback, and speak() recorded success while nothing was ever heard — voice went permanently dark until a restart. Availability now needs an explicit opt-in signal (env var set, CLI engine present, or a provider-registry health check that actually probed the server), and a failed neural synth falls back to browser speech with a warning instead of silence. - The warm-piper HTTP path now counts toward piper availability (PIPER_HTTP_URL set), instead of demanding the CLI + model that the warm server replaces. - Selecting TTS 'none' in the provider registry now actually mutes TTS — applyActiveTts previously did nothing on null, leaving the old backend speaking. - priority now survives speakLocally() into the streamed-audio path, so a high-priority clip can interrupt again. - client: streamed audio respects the mute switch, an autoplay-blocked clip is replayed on the first user gesture instead of being lost, and consecutive clips queue instead of talking over each other. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… public repo The earlier scrub covered the example manifest and the public skill, but this PR still shipped real private repo names (with quality judgements attached) in the new design doc, the new Commander-doc examples, and the atlas test fixtures. All replaced with the placeholder vocabulary the scrubbed files already use (acme-*, physics-kit, puzzle-proto, example-engine). Pre-existing references on main (docs/COMMANDER_CLAUDE.md worktree examples, older PLANS/) are deliberately untouched — that exposure predates this PR and is a separate decision. Note the scrubbed names do remain in this branch's earlier commit history. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion
The negation guard matched any utterance starting with stop/cancel, so a
valid command that missed an exact rule phrasing ('stop the server') was
short-circuited away from the LLM classifier and misrouted to the
Commander. Only the dismissal forms ('stop that', 'cancel it', 'forget
that') short-circuit now; don't/never/no/nope still guard as before.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… grounding + capture
- CRITICAL: rule matching ran BEFORE the negation guard, and many rule
patterns are unanchored substrings — verified live, "don't stop all
claudes" matched stop-all-claudes and executed it fleet-wide with no
confirmation gate ('don't kill work 3' likewise). Negation is now
detected first and skips rule matching entirely; the utterance goes to
the Commander, which understands "don't".
- /what.*sessions/ was greedy enough to steal natural fact questions
("what are my sessions doing") from the fact lane, speaking a useless
"Done — list sessions." instead of the real fleet status. Only
enumerative phrasings match the rule now.
- isGrounded accepted a hallucinated destructive command if ANY token
matched — "is my session about to time out" grounded kill-session via
the word 'session'. A destructive command now requires its verb to have
actually been heard.
- Commander-reply captures are serialized: two concurrent polls over the
one shared PTY buffer could speak one utterance's answer for the other.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ooldowns; audit rule reloads
The auto-approve deny list was bypassable three ways, all reproduced
against the live loaded rules:
- Fully case-sensitive matching: Write(.ENV) and lowercase SQL (drop
table / delete from / truncate) evaded denies written as \.env and
\bDROP\b. Deny patterns now compile case-insensitive (fail-closed
direction only — allow patterns stay exact so nothing NEW auto-approves).
- The credentials deny only knew the ~/ form; Write(/home/x/.ssh/...) —
the way agents actually render out-of-repo paths — was auto-approved.
Absolute /home, /Users and /root forms are now denied too.
- Quoted paths (Write("package.json")) slipped the exec-on-next-op
anchors; quote/space/equals now count as boundaries, and git push -f
is denied like --force.
Also: forgetHealed() never cleared cooldowns, so a stale timestamp from
an already-healed occurrence silently blocked action on a genuinely new
recurrence (and the map grew unbounded); reload-rules now writes an
audit entry like every other config-changing action.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ted channels, awaited task records - Permalinks always rendered the DM-only /channels/@me/... form: REST message objects don't carry guild_id and nothing looked it up. The guild id is now fetched once per channel (cached in state), so links into guild channels — the primary use case — actually work. - A first-sight backfill paged FORWARD from the newest message, so any backfillMessages > 100 silently capped at 100 (the second page asked for messages after the newest and got nothing). Backfills now walk before= into history and keep the most recent N. - addChannel/removeChannel only mutated memory; a channel added over the API vanished on reload-config or restart. Both now persist to the override config file the reload reads. - linkSession's task-record upsert is async but wasn't awaited — its try/catch was dead code and a failure became an unhandled rejection. - numberOr(null|''|boolean) now falls back instead of coercing to 0 (a null minLength disabled the spam filter), and an explicit tier: 0 is honored instead of ||-coerced to 3. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JARVIS: an assistant that fixes things itself and only surfaces what it genuinely couldn't handle.
Design + research:
PLANS/2026-07-26/AUTONOMOUS_ORCHESTRATOR_VOICE_AND_REPO_ATLAS.md,PLANS/2026-07-26/RESEARCH_HERMES_CODEX_AND_DISCORD.md.1. Supervisor — fix first, interrupt last
A finding cannot reach you until it has exhausted its repair attempts, cleared an urgency threshold weighted by the task's tier, and fitted inside an interruption budget. Everything else self-heals silently or batches into a digest.
continue, and never tells you — waiting isn't a problem you need to know about.Default autonomy is
autopilot. Defensible because: named handlers only (rules can't inject shell), fail-closed permission approval refusing credentials/force-push/merge/rm -rf, full audit trail.2. Codex app-server — facts instead of scraping
openai/codexis Apache 2.0 and the CLI already installed shipscodex app-server. Built and verified against the real process —initializehandshake, livethread/list(which returnsdata, notthreads; the schema names imply otherwise), live notification flow.ThreadActiveFlag: waitingOnApprovalturn/completedwith status + durationthread/status/changedsystemErrorthread stateTwo conditions use these:
structured-approvalfires in seconds because a reported wait needs no quiet-time threshold to be believed, andthread-system-errordelegates a runtime failure rather than nudging it. Approvals are answered over the wire with the command in hand.Degradation is the design —
getSignalForSessionreturning null means "no better information than the PTY", so Claude/Gemini/aider are untouched. Opt in withCODEX_APP_SERVER=true.3. Repo Atlas — git-backed, shareable, self-updating
One file per repo under
entries/— two machines curating different repos never conflict. Verified against a real remote with interleaved edits.Write-back: agents propose highlights with evidence; you approve. They can't write directly — left free, every repo an agent touched would end up 5/5 and the scores would stop meaning anything.
atlas subscribereads a teammate's bundle, attributed, lowest precedence, provably un-re-shareable.4. Discord — ambient, durable, publishes status
linkSession()binds an item to the session doing it and announces it in-thread. "Is their agent working on it" stops being a question, and/untrackedis the list that until now lived only in scrollback.5. Voice + UI
Full-duplex loop against Codex threads: browser STT →
thread/realtime/appendText→ transcript deltas → spoken back. Only completed assistant turns are spoken (browser-verified: partial deltas correctly ignored). Unmatched speech anywhere goes to the Commander, so the fallback is an agent, not an error beep.Alt+J opens the JARVIS panel: what was handled, what needs you, untracked chat work, Atlas proposals with one-click approve, Atlas search.
End-to-end verification (2026-07-27)
Driven live against a running server with
CODEX_APP_SERVER=true+autopilot, an isolated HOME, and a real bare-git atlas remote. Every new endpoint hit with valid AND invalid input (82/82, zero 5xx, zero uncaught errors over the whole run), plus real integration flows. Six more bugs found and fixed this way, none catchable by unit tests alone:exitevent clobbered its replacement (rejected the new initialize, nulled the ref, spurious respawn); and the handshake only ran instart(), so any auto-restart came back uninitialized. Handlers now bind to their own child and bail once replaced; the service re-handshakes on every spawn.0; the HTTP path param is the string"0"— the Map lookup missed and every answer failed. Verified end-to-end by driving a real Codex thread to a command-approval prompt and approving it over the wire (thread ran to completion).atlas.config.json(remote URL, local output paths) was inside the synced tree. Now ignored + self-heals existing registries. Full A→remote→B→remote→A judgement round-trip now clean.turn/completed+ token usage; publish→subscribe with a third machine provably unable to re-share; browser pass with seeded Discord items (XSS payload rendered as inert text), one-click proposal approve writing through to the registry, atlas search, and speech-out — zero console errors; 90s soak with everything on stayed healthy with no orphaned processes.Suite now 835 green / 118 suites.
Research verdicts
Hermes Agent — no. Can't run on the Codex subscription (wants an OpenAI-compatible endpoint; the CLI isn't one), and its runtime duplicates ours without knowing worktrees or tiers. Its messaging gateway and self-improving skill loop are worth stealing as patterns.
Codex app — beaten, not copied. Theirs is Codex-only and macOS-only. Speaking app-server for Codex while keeping PTY scraping universal covers more ground than either product.
Test plan
npm run test:unit— 815 passing, 117 suites (652 on main).codex app-server: handshake, thread list, notifications.Number(null) === 0bug class silently zeroing unset numeric config.Honest limits
Raw-audio realtime (
appendAudio) is wired but its PCM framing is unverified against a live authenticated session — text path is the default and works. Structured signals cover Codex only. Discord extraction is rules-only; the model seam exists but is unused.🤖 Generated with Claude Code
Review pass (fixes on this branch)
A multi-agent read-only review of every subsystem, plus a mine of the creation-session log, turned up a set of real bugs — all now fixed on this branch, tested, and verified (full suite 829 green, up from 815; live boot with
CODEX_APP_SERVER=true+SUPERVISOR_AUTONOMY=autopilotagainst the realcodex app-server, no crashes, no orphaned processes).Critical
errornotification (a real type) or any child-process error was re-emitted as the reserved EventEmittererrorevent with no listener, crashing the whole orchestrator via uncaughtException; andstart()cached a stale resolved promise so auto/manual restart never respawned.PLANS/anddocs/historical/already reference these names — a separate, pre-existing exposure to decide on.)autopilotdefault, the Edit/Write allowlist auto-approved writes to.git/hooks,package.json,.github/workflows, shell rc files, etc., which then execute via the already-allowlistednpm run build/git commit. Added path-based deny patterns (fail closed).Correctness / durability
quality: null) were forced to quality 1; subscribed repos became re-shareable the moment you annotated them; non-atomic registry writes could commit a corrupted half-write.NaN;"not done yet"closed items; first-sight backfill kept the oldest N instead of the most recent N; non-atomic state writes.--flag=valuetruncated values containing=; a value-less--qualitysilently became 1.Second pass (scout reports cross-checked): proposals-file atomic write, app-server buffer overflow no longer drops queued frames, supervisor nudge conditions require
agentPresent(no typing into dead shells regardless of rule ordering), tier||nullgotcha, voice status no longer shows "null". Design doc + codebase map corrected to the shippedautopilotdefault and real ladder semantics. UI verified in a real browser under Xvfb: Alt+J opens/closes, is suppressed while typing, action failures surface in the error banner, and the realtime thread filter provably ignores other threads' transcripts and close events.Consciously left as-is (documented, not bugs): unmatched-voice→Commander forwarding (the headline feature), the app-server thread-id linkage (dormant, degrades to PTY — now stated in the design doc), and the rules-only Discord/delegate prompt-injection surface inherent to the design.
Review pass 3 (Fable, 2026-08-05)
A second full multi-agent review (7 read-only subsystem scouts + an independent Codex pass over the voice commits that landed after review pass 2), every finding verified against the code before fixing. Suite now 893 green / 120 suites (was 868). Ten commits.
Critical
stop-all-claudesfleet-wide with no confirmation gate. Negation is now detected first and skips rule matching entirely.Write(.ENV), lowercasedrop tablepassed); credential denies only knew the~/form while agents render absolute paths (Write(/home/x/.ssh/authorized_keys)was auto-approved); quoted build-file paths slipped the exec-on-next-op anchors. Deny patterns now compile case-insensitive (deny-only — nothing new auto-approves), absolute credential paths andgit push -fare denied.speak()reported success. Honest availability signals + browser-speech fallback when neural synth fails.foreigncleared as soon as discovery also knew the repo, so a teammate’s highlights republished in your own bundles. Bundles now compile from your own layers only — subscription content structurally cannot leave.Correctness / durability
/channels/@me/…(dead links — guild id now fetched + cached); backfills above 100 silently capped (pagination went the wrong direction); API-added channels didn’t persist;linkSession’s task-record write wasn’t awaited (dead catch);numberOr(null)→ 0;tier: 0coerced to 3./what.*sessions/stole natural fact questions from the fact lane;isGroundedaccepted hallucinated destructive commands off an incidental noun; concurrent Commander-reply captures raced on the shared buffer (now serialized).nonedidn’t mute; priority was dropped on the streamed-audio path; the client played overlapping clips, ignored the mute switch for streamed audio, and lost autoplay-blocked utterances (now queued/respected/replayed on first gesture).atlas initwrote the manifest outside the repo formain/worktree layouts; value-less--topicrecorded the literal topic “true”.reload-ruleshad no audit entry.Privacy
Consciously left as-is:
POST /api/atlas/proposalsstays read-level (agents must be able to propose; approving is what writes and stays write-gated);speech-*socket events are fleet-wide broadcasts by design;extractAssistantReplyremains heuristic (the structured replacement is the app-server path);engines >=16predates this PR though the real floor is Node 18 (fetch).🤖 Generated with Claude Code