Skip to content

3zrv/zoigraph

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

24 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

zoigraph

A native, 3D force-directed personal knowledge base and situational-awareness map. Static nodes live in a local SQLite database; ephemeral phantom nodes are injected over a loopback UDP socket and decay over a TTL. Edges carry a label, a kind, and a certainty that drives their render alpha. Every project is its own DB file under projects/.

A built-in LLM bridge lets a local model (Ollama by default) propose phantoms about the selected node. A structured trust gradient — node tier + edge certainty + UDP-only injection — keeps model output visually marked as provisional until a human pins it, and every pin / decay / edit / delete / bones-throw is logged to a per-project events table for analysis.

Built for local-only (air-gapped) use: on the default path nothing leaves the machine. The repo also ships an optional, non-default claude backend for the LLM bridge (CLI --backend claude, off unless you ask for it) that sends prompt context to Anthropic's API — see LLM bridge. At-rest encryption (SQLCipher) is on the roadmap, not yet a delivered guarantee.

zoigraph.png CTEacCX.md.png

work in progress

Build

CMake ≥ 3.24 and a C++20 compiler. On Linux you also need X11/Wayland + OpenGL development headers for raylib:

sudo apt install build-essential cmake \
    libgl1-mesa-dev libx11-dev libxrandr-dev libxinerama-dev \
    libxi-dev libxcursor-dev libxext-dev libwayland-dev libxkbcommon-dev

macOS and Windows need nothing past their default toolchain (Xcode Command Line Tools / MSVC). Configure and build:

cmake -S . -B build
cmake --build build -j$(nproc)         # Linux/macOS
cmake --build build --config Release    # Windows
./build/zoigraph                        # or build\Release\zoigraph.exe

The first configure downloads pinned versions of raylib 5.5, Dear ImGui v1.92.8, rlImGui, the SQLite amalgamation 3.45.2, nlohmann/json 3.11.3, and doctest v2.4.11 via FetchContent — two to three minutes online, cached locally after that.

Sanitizer build (AddressSanitizer on every translation unit — the third-party deps too, since partial instrumentation breaks linking):

cmake -S . -B build-asan -DZG_ASAN=ON

Controls

input effect
left-click a node select
double-click select + open the Inspector tab
right-drag orbit camera
shift + right-drag pan
scroll wheel zoom
R reset view
H rabbit hole (3-hop fly from select)
B throw the bones (3 weakly-linked)
T timeline collapse / restore
L toggle all node titles overlay
ESC × 3 clean exit

Node titles auto-render on the selected node + its 1-hop neighbours + every phantom's connection targets + the bones triple. Press L to flood the field with every non-deleted title for whole-graph navigation.

The control panel has five tabs:

  • Project — switch / create / delete projects, tag filter (with dim-non-matching toggle), auto-cluster, and view flags (grid, CRT shader, Barnes-Hut physics). Every checkbox has a hover tooltip.
  • Inspector — node/edge/phantom/fps stats, an FTS5-backed real-time search box (jumps camera + selection to the top hit), the selected-node editor (tier, tags, markdown content with [[wikilinks]]), and the incident-edges editor (label / kind / certainty) plus add-edge-by-title-search. "ask about selection" fires the configured LLM at the selected node and lets the result land as a phantom; "delete node…" soft-deletes (two-click arm/confirm) so the node vanishes from the field without losing its pin trace in events.
  • Toolbar — create a static node (optionally spawned beside and edged to the current selection), inject a phantom locally, or save a timestamped journal entry (auto-edged from self and from the current selection).
  • CLI — a claude-style slash-command prompt with scrollback (/help lists everything; Up/Down recalls history, Tab completes, /clear wipes the pane). See below.
  • Help — a terse cheat-sheet of every input.

CLI commands

  • /node "title" [--tier t] [--to ref] / /edge <a> <b> [kind] — create nodes and edges; a node ref is an id, an exact title, or an FTS search term.
  • /select <ref> / /neighbors <ref> — select + fly to a node; list a node's incident edges with kind + certainty.
  • /tier <ref> <t> — set a node's tier (confirmed / suspected / phantom).
  • /delete <ref> [--confirm] — tombstone a node; the unarmed form prints the exact confirming command with the resolved numeric id, so a fuzzy ref can't delete something other than what it showed.
  • /projects, /project <name> [--create], /info — list / switch projects and show active-project stats.
  • /search <terms…> — FTS5 search; lists hits, selects + flies to the top one (shares state with the inspector search box).
  • /phantom "label" [--n N] [--cat c] — inject a batch of local phantoms; /phantoms [cat] lists active ones; /filter <cat|all> hides phantoms outside one category (visual only — hidden phantoms still age, decay, and log).
  • /pin <phantom-id> / /decay <phantom-id|all> — accept (promote to a static node, same path + event log as click-to-pin) or dismiss without pinning (logs an ordinary decay at dismissal time).
  • /ask [ref] — fire the LLM bridge at a node (defaults to the selection); resulting phantoms land via the UDP listener.
  • /settings, /set <grid|crt|dim> <on|off>, /set port <n> (or /port <n>), /set size <WxH> — view, listener, and window-size settings, persisted in settings.json. A port change tears down and rebinds the UDP listener in flight; a size change resizes the window live. Window size also follows drag-resizes — the last size is captured on clean exit and restored next launch.
  • /panic overwrites then deletes all project data (every projects/*.db with WAL/SHM sidecars, the .last marker, settings.json, and any leftover legacy zoigraph.db) and exits. The overwrite-before-unlink is best-effort — CoW filesystems, SSD wear-levelling, and journals can retain stale blocks; real at-rest protection arrives with SQLCipher.

Telemetry — phantom injection

A UDP listener is bound to 127.0.0.1:7777 (loopback only; port configurable via /set port <n>, persisted in settings.json). Each datagram is parsed as one phantom-node JSON payload:

{
  "id": 42,
  "x": 5.0, "y": 5.0, "z": 5.0,
  "label": "scan-host-1.2.3.4",
  "content": "one or two sentences of reasoning",
  "source": "ollama:llama3.2:3b",
  "category": "recon",
  "connections": [
    {"target": 1,  "kind": "saw-at"},
    {"target": 19, "kind": "shell-of"}
  ]
}

Every field except id / x / y / z is optional. connections also accepts the legacy shape [1, 7, 19] (bare ints) — each becomes a connection with empty kind. Quick test from the shell:

echo -n '{"id":42,"x":5,"y":5,"z":5,"label":"hi"}' \
    > /dev/udp/127.0.0.1/7777

The phantom appears as an additive-blended glowing wireframe sphere and decays to alpha 0 over a 60-second TTL. Click-to-pin promotes it to a permanent static node at the lowest tier="phantom", with its edges at certainty="phantom" (visibly faded until the operator promotes them in the inspector). Payloads are capped at 1 KiB; over-sized datagrams are dropped, not parsed-truncated.

This is a write-only channel. The complementary read channel — how the LLM bridge asks the live graph for context — is the query channel.

LLM bridge

The inspector's "ask about selection" button is the in-app path: it shells out to scripts/llm_phantom.py emit --anchor-id <id>, which pulls the selected node's relevance neighbourhood from the running app over the query channel (below), builds a prompt from it, and lets the response land via the normal UDP listener. A 100 ms TCP probe against 127.0.0.1:11434 runs first, so a missing Ollama daemon surfaces a red error inline rather than hanging. Backend is hardcoded to ollama:llama3.2:3b for now.

External agents can drip phantoms from any source:

python3 scripts/llm_phantom.py emit \
    --backend ollama --model llama3.2:3b \
    --db projects/<active>.db \
    --anchor-id <node-id>

The script enforces a strict JSON schema before sending and tags every emission with source="<backend>:<model>", so the analysis can break pin rate down by emitter (the ceiling-vs-floor comparison between models).

Backends (--backend): mock (synthetic JSON, no model needed), ollama (local — the default, and the only backend the in-app button uses), and claude. The claude backend is remote: it POSTs the prompt (your selected node's title plus a short content snippet of each neighbour) to Anthropic's API. It is opt-in only — never the in-app default, selected explicitly on the CLI, and requires ANTHROPIC_API_KEY in the environment (no key is bundled). It exists for the phase-2 ceiling comparison; per the project's sensitivity rule, point it only at benign corpora, never at real or sensitive graph content.

Query channel

A second loopback listener — a UDP request/response channel on 127.0.0.1:7778 (configurable via query_port in settings.json) — lets the bridge read the live graph instead of opening the SQLite file directly. Three read queries, one JSON object per datagram:

  • {"q":"neighborhood","id":<id>,"hops":<n>,"token":"…"} — the anchor plus its most relevant neighbours, ranked by personalized PageRank (graph relevance, not raw spatial proximity), with the edges among them.
  • {"q":"search","text":"…","token":"…"} — FTS5 hits.
  • {"q":"node","id":<id>,"token":"…"} — one node.

Replies exclude tombstoned nodes and truncate each node's content so they fit a single datagram. The render thread answers from its own data once per frame; the socket thread never touches the graph. Each request must carry a per-session token the app writes (mode 0600) to <db>.db.token on open — the read channel exposes node content, so an unauthenticated caller is dropped with no reply. It also keeps the eventual SQLCipher swap-in clean: the emitter never needs the DB key. emit falls back to a direct DB read when the channel is unreachable (no running app, or --no-channel).

Events table

Every phantom-lifecycle event lands in a per-project events table:

kind when
phantom_spawn UDP packet parsed + added to the buffer
phantom_pin operator click-to-pin promotes to a static node
phantom_decay TTL expired without a pin
node_edit inspector text-edit on title/content
node_delete inspector soft-delete
bones_throw B key triggers a 3-node bones throw

Dump and summarise after a session:

python3 scripts/export_events.py --db projects/<active>.db

Output includes total pin rate, time-to-pin distribution, pin-then-edit-within- 60s rate, a per-source breakdown, and a stop-criterion check (pin rate should sit in [5, 50] for the trust gradient to be doing real work).

Performance & scale

zoigraph is tuned for the hundreds-to-low-thousands of nodes a hand-curated graph actually reaches — there, every subsystem is sub-frame and the layout settles in well under a second. It opens far larger graphs too: node bodies draw in a single instanced call, and physics freezes on convergence (it stops integrating once the layout settles and wakes only on a change), so a big, settled graph costs no CPU.

The wall for live interaction at scale is physics throughput: a Barnes-Hut tick is roughly 25 ms at 10K nodes, 0.4 s at 100K, and 2.8 s at 500K, so a 500K graph opens and renders but lays out very slowly. Two render paths also degrade past ~100K because they are immediate-mode per node — the auto-cluster halos and the L all-labels overlay — so leave those off on huge graphs.

tools/bench_scale is a headless probe for these numbers (built on demand):

cmake --build build --target bench_scale
./build/bench_scale 10000 100000 500000

It can also write a synthetic project DB for eyeballing the real app at scale:

./build/bench_scale --write-db projects/big.db 100000

Tests

(cd build && ctest --output-on-failure)

Twenty-eight doctest binaries: forces, integrator, barnes_hut, graph_buffer, picks, cluster, ppr, timeline, wikilinks, escape_wipe, db, project_store, secure_wipe, seed, settings, phantom, query_protocol, query_responder, query_token, ask, promote, phantom_lifecycle, labels, cli, session, paths, rabbit, plus a placeholder sanity check.

Pure-logic modules ship with doctest cases before being threaded into runtime code; render-loop and ImGui-bound code is exempt by design (no useful unit-test path without a window).

Architecture

Four threads, no IPC, no fibers, no third-party concurrency lib:

  1. Main / render — raylib window, Camera3D, instanced node draw via a GLSL 330 vs/fs pair (one draw call for the whole field, split in two when a tag filter dims non-matching bodies), a CRT post-process pipeline (chromatic aberration + scrolling scanlines + vignette over an off-screen render texture), and the ImGui inspector frame layered on top. Also drains the query channel and answers it from its own graph data.
  2. Physics — Coulomb pairwise repulsion + Hooke springs along edges + a linear centering force; symplectic Euler with damping and a velocity clamp; positions published at 120 Hz through a mutex-guarded GraphBuffer. Toggleable O(N²) ↔ Barnes-Hut octree. Freezes on convergence: once the graph's RMS node speed settles below a threshold the thread pauses the (expensive) integrate step until something perturbs it — a new node/edge, a pin change, or a live phantom — so a settled graph costs no CPU and a big one stops burning Barnes-Hut ticks forever.
  3. Telemetry — UDP socket polled on a 100 ms tick (POSIX poll, WSAPoll on Windows); JSON → Phantom pushed into a TTL-expiring PhantomBuffer.
  4. Query channel — sibling UDP request/response socket on :7778 (same platform shim, recvfrom/sendto). The socket thread only moves bytes through a mailbox; the render thread drains it once per frame and answers reads (PPR neighbourhood / FTS search / node) from its own data.

Persistence is plain SQLite (with FTS5) at projects/<name>.db. The schema covers nodes (with tier + soft-delete tombstone), edges (label / kind / certainty), node_tags, meta (project-level key/value), and events (the append-only telemetry log). The persistence layer is structured for a SQLCipher swap-in: the linked target is named sqlite3 so the symbol surface stays identical when AES-256-GCM lands. Node identity is id == vector index; the app refuses to open a DB whose ids aren't contiguous 0..N-1 rather than silently mis-pointing edges.

Platforms

A single binary on each:

  • Linux — primary development target.
  • macOScmake -S . -B build && cmake --build build Just Works.
  • Windows — same. The sockets use Winsock2 behind an #ifdef _WIN32; CMake links ws2_32 instead of pthread.

CI runs the build + test matrix across all three on every push; tagged releases (v*) ship stripped binaries to a GitHub Release. See .github/workflows/.

Layout

src/
├── main.cpp                          # ~520-line render-loop wiring
├── app/                              # session state, hotkeys, render-loop glue
│   ├── session.{h,cpp}               # per-project Session struct + open_project
│   ├── hotkeys.{h,cpp}               # ESC/H/B/T/L key handlers
│   ├── pick.{h,cpp}                  # mouse raypick + click-to-pin
│   ├── pin.{h,cpp}                   # promote a phantom to a static node
│   ├── promote.{h,cpp}               # pure phantom→StoredNode promotion (tested)
│   ├── phantom_lifecycle.{h,cpp}     # per-frame spawn/decay diff (tested)
│   ├── settings.{h,cpp}              # persisted operator settings (tested)
│   ├── paths.{h,cpp}                 # data-dir + resource path resolution (tested)
│   ├── clock.h                       # monotonic clock shim (header-only)
│   ├── ask.{h,cpp}                   # LLM Ask button: TCP probe + popen
│   ├── query_responder.{h,cpp}       # answer channel reads from the live graph (tested)
│   └── query_token.{h,cpp}           # per-session 0600 auth token (tested)
├── graph/                            # types, thread-safe buffer, pure algos
│   ├── types.h                       # shared Edge struct (label / kind / certainty)
│   ├── graph_buffer.{h,cpp}          # mutex-guarded positions handoff
│   ├── picks.{h,cpp}                 # weakly-connected triple picker
│   ├── cluster.{h,cpp}               # label-propagation
│   ├── ppr.{h,cpp}                   # personalized PageRank for relevance (tested)
│   ├── timeline.{h,cpp}              # first_seen → y-z spiral disc layout
│   └── wikilinks.{h,cpp}             # [[title]] parser
├── input/escape_wipe.{h,cpp}         # triple-ESC-to-exit state machine
├── macros/                           # operator-triggered camera flies
│   ├── rabbit_hole.{h,cpp}
│   └── bones.{h,cpp}
├── persistence/                      # SQLite + per-project files
│   ├── db.{h,cpp}                    # schema, FTS5 triggers, events log, id check
│   ├── project_store.{h,cpp}         # list / create / delete projects
│   ├── secure_wipe.{h,cpp}           # /panic overwrite-then-unlink wipe
│   └── seed.{h,cpp}                  # fresh-project seed + bulk-fill generator
├── physics/
│   ├── forces.{h,cpp}                # pure Coulomb + Hooke + convergence metric (tested)
│   ├── barnes_hut.{h,cpp}            # octree repulsion (tested)
│   └── physics_thread.{h,cpp}        # integrate_step + freeze-on-convergence (Thread 2)
├── render/                           # GPU + 2D overlay code
│   ├── shaders.h                     # GLSL 330 instancing + CRT fragment
│   ├── imgui_theme.{h,cpp}
│   ├── camera.{h,cpp}                # orbit camera + defaults
│   ├── draw.{h,cpp}                  # draw_jagged_line + small helpers
│   ├── scene.{h,cpp}                 # full 3D pass (bodies/edges/halos)
│   ├── composite.{h,cpp}             # CRT composite + edge labels + ESC HUD
│   ├── labels.{h,cpp}                # in-focus title set + DrawText overlay
│   └── sizes.h                       # kNodeRadius / kPhantomRadius
├── telemetry/                        # Thread 3 inject listener + Thread 4 query socket
│   ├── phantom.{h,cpp}               # Phantom + Connection structs
│   ├── phantom_parse.{h,cpp}         # JSON → Phantom (both connection shapes)
│   ├── phantom_buffer.{h,cpp}        # TTL-expiring shared buffer
│   ├── telemetry_thread.{h,cpp}      # :7777 UDP phantom inject listener
│   ├── query_protocol.{h,cpp}        # query-channel wire format (tested)
│   └── query_thread.{h,cpp}          # :7778 request/response socket + mailbox
└── ui/                               # ImGui tab/panel renderers
    ├── project_tab.{h,cpp}
    ├── inspector_tab.{h,cpp}
    ├── toolbar_tab.{h,cpp}
    ├── cli_tab.{h,cpp}               # slash-command prompt (/panic, /help)
    ├── help_tab.{h,cpp}
    └── bones_panel.{h,cpp}

tools/
└── bench_scale.cpp                   # headless N-scaling probe (EXCLUDE_FROM_ALL)

scripts/                              # external bridge + analysis
├── llm_phantom.py                    # backend-agnostic emit + measure harness
├── seed_corpus_unix.py               # UNIX-history sample corpus seeder
└── export_events.py                  # events table → CSV + pin-rate summary

tests/                                # one doctest binary per module

License

Copyright (C) 2026 Mohamed Sayed.

zoigraph is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. It is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the full text in LICENSE.

Third-party dependencies are vendored at build time via FetchContent under their own permissive licenses (zlib / MIT / public domain) — see THIRD_PARTY_LICENSES.md. Their notices are bundled into every release binary.