Callisto

A real-time voice AI assistant with raw audio streaming over WebSocket, featuring seamless conversational flow, AI-driven orb visualization and browser-integrated tool execution.

---

What makes it interesting

Most "AI voice assistants" are speech-to-text → LLM → text-to-speech, with a second or two of latency at each hop. Callisto streams raw audio in both directions over a single WebSocket to the Gemini Live API, so there is no transcription round-trip in the critical path — the model hears audio and emits audio.

• Barge-in works. Speaking over Callisto stops her playback immediately, because the microphone stream never pauses.
• The orb is driven by the actual signal — volume and a spectral-centroid pitch proxy sampled from both audio graphs at 20 Hz, not a timer.
• Tools round-trip through the browser. One of them holds the model's turn open until the browser answers.

Read the architecture →

Quickstart

Prerequisites: Node 22+ and a Gemini API key.

git clone https://github.com/nelay04/Callisto.git
cd Callisto
npm install

cp apps/server/.env.example apps/server/.env     # add your GEMINI_API_KEY
cp apps/web/.env.example    apps/web/.env

npm run dev

Open http://127.0.0.1:3012, allow microphone access, and start talking.

Prefer containers? Both stacks read those same two .env files, plus a root .env holding the two build-time NEXT_PUBLIC_* URLs:

cp .env.example .env     # optional locally; defaults to loopback
npm run docker:up        # or: npm run podman:up

Full running guide — bare, Docker, Podman →

Documentation

Architecture	How it fits together, the audio pipeline, tool calling, limitations
Running	Bare npm, Docker and Podman, plus troubleshooting
Configuration	Every environment variable, and the system prompt
Deployment	nginx, a domain, TLS, what to expose, and the deploy workflows
Contributing	Tests, conventions, manual QA pass

Repository layout

callisto/
├── apps/
│   ├── server/          Express + WebSocket bridge to Gemini Live
│   └── web/             Next.js 15 client with the reactive orb
├── packages/
│   └── protocol/        Shared WebSocket message types + validators
└── docs/core/           Architecture, running, configuration, deployment

Workspace	Stack
@callisto/server	Node 22, Express 4, ws, @google/genai
@callisto/web	Next.js 15, React 19, Tailwind 4, Motion
@callisto/protocol	Zero-dependency TypeScript

Ports

	Port	Bound to
@callisto/web	3012	127.0.0.1
@callisto/server	3013	127.0.0.1

Everything listens on loopback rather than 0.0.0.0, in every mode — the compose files publish as 127.0.0.1:3012:3012, so nothing is reachable from the network. In production a single nginx block fronts both on one domain; see deployment.md.

Development

npm run dev         # both apps, with hot reload
npm run typecheck   # tsc --noEmit across all workspaces
npm run lint        # ESLint
npm test            # Vitest
npm run build       # production build of all three workspaces

Tests cover the pure logic where a silent bug would be hardest to spot by hand: the PCM16 encode/decode round-trip, transcript-turn merging, protocol validation, WebSocket admission control, and system-prompt loading. Audio playback and the Gemini session itself are verified by hand — see CONTRIBUTING.md.

Every workflow is manual, started from the Actions tab. CI runs the commands above plus a container build and a compose parse; Deploy (Docker) and Deploy (Podman) ship main to the VM over SSH. Nothing runs on push, and merging never deploys on its own — see deployment.md.

Making it yours

Callisto's persona is deliberately specific — she's named for both Jupiter's outermost Galilean moon and the Greek huntress, and she introduces herself as a portfolio companion.

Her entire personality is one environment variable, CALLISTO_SYSTEM_PROMPT in apps/server/.env. Rewrite it, set your own CALLISTO_LINKS and MAILTO_ADDRESS, and restart — no recompile and no image rebuild. See configuration.md for the multi-line formatting rules.

License

MIT © Nelay Karmakar