Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ All notable changes to this project are documented here. The format is loosely b

### Fixed
- **GitHub-monitor cron no longer blocked by invisible unicode.** `scripts/stack_monitor.py` fetches upstream GitHub release names and commit messages, which routinely embed zero-width / invisible "format" (Cf) characters — most often the ZWJ `U+200D` inside emoji sequences like 👨‍💻. When the JSON report was fed back into Hermes to format for Discord, that invisible unicode tripped the prompt-injection scanner and the whole daily cron failed (`Blocked: prompt contains invisible unicode U+200D`), every run. The monitor now strips all Cf-category characters from the entire output (recursively, both `--json` and human-readable) before emitting; visible text and emoji are unaffected (a ZWJ emoji renders as its component glyphs). Covered by `tests/test_stack_monitor_sanitize.py`.
- **ComfyUI restart storm hardened (status verb + server-side debounce).** During a failed reel cron, Hermes tried to recover ComfyUI by hammering guessed paths `POST /api/comfyui/restart` and `GET /api/comfyui/status` against both ops-controller (404) and the dashboard (401), ~every 3s for ~2 min — it had no canonical *status* verb so it improvised raw HTTP. Two fixes: (1) ops-controller `POST /services/{id}/restart` now **debounces** rapid repeats per-service (`OPS_RESTART_DEBOUNCE_SECONDS`, default 20) so a retry-loop collapses into one in-flight restart instead of stacking `docker restart` calls; (2) a new ComfyUI-**independent** health verb — dashboard `GET /api/orchestration/comfyui/status` + orchestration-MCP `comfyui_status` tool — that reports container state + render-queue reachability by querying the dashboard→ops control plane (which stays up when ComfyUI is down), so the agent checks status via a real tool instead of guessing. The `restart_comfyui` tool docstring now points agents at `comfyui_status` rather than tight-looping.
- **ComfyUI restart storm hardened (status verb + server-side debounce).** During a failed render cron, Hermes tried to recover ComfyUI by hammering guessed paths `POST /api/comfyui/restart` and `GET /api/comfyui/status` against both ops-controller (404) and the dashboard (401), ~every 3s for ~2 min — it had no canonical *status* verb so it improvised raw HTTP. Two fixes: (1) ops-controller `POST /services/{id}/restart` now **debounces** rapid repeats per-service (`OPS_RESTART_DEBOUNCE_SECONDS`, default 20) so a retry-loop collapses into one in-flight restart instead of stacking `docker restart` calls; (2) a new ComfyUI-**independent** health verb — dashboard `GET /api/orchestration/comfyui/status` + orchestration-MCP `comfyui_status` tool — that reports container state + render-queue reachability by querying the dashboard→ops control plane (which stays up when ComfyUI is down), so the agent checks status via a real tool instead of guessing. The `restart_comfyui` tool docstring now points agents at `comfyui_status` rather than tight-looping.
- **Dashboard at `/dash/` renders fully behind SSO front door (#14).** Three coupled bugs prevented the dashboard from working when served at `/dash/`: (a) all `fetch('/api/...')` calls used absolute paths and missed Caddy's `/dash/*` prefix → `index.html` now detects `APP_PREFIX` from `location.pathname` and prepends it inside the `api()` wrapper, the bootstrap `/api/auth/config` fetch, and the favicon link; (b) service-card links pointed at direct upstream ports (e.g. `:8188`) that aren't host-published — `serviceOpenHref` now maps known SSO-gated services (`comfyui` → `/comfy/`, `webui` → `/chat/`, `n8n` → `/n8n/`, `hermes` → `/hermes/`) to their Caddy paths when `APP_PREFIX` is set; (c) `/api/auth/config` advertised `auth_type=bearer` even when the request arrived from the trusted reverse proxy with a verified `X-Forwarded-Email` → endpoint now returns `auth_required=false` on the SSO path so the bearer-token modal doesn't pop up.
- **Caddy `route { forward_auth + handle_path }` pattern restored on main.** The original `handle @auth { forward_auth }` shape was terminal — once `forward_auth` returned 202, the request ended without ever reaching the path-prefix handler, so SSO-gated routes (`/dash/`, `/chat/`, `/n8n/`, `/hermes/`, `/comfy/`) returned an empty 202 instead of the upstream UI. Pairs with the `--email-domain=*` removal in oauth2-proxy (the wildcard bypassed the `--authenticated-emails-file` allowlist).
- **Caddy moved off `frontend` network.** Membership on both `proxy-net` and `frontend` made Docker DNS return whichever IP it picked first for upstream services. When it chose the `frontend` IP, dashboard saw caddy as outside `DASHBOARD_TRUSTED_PROXY_NET=172.24.0.0/16`, refused the `X-Forwarded-Email` header, and 401'd every `/api/*` call. Caddy is the SSO ingress only and reaches every upstream over `proxy-net`; dropping `frontend` membership eliminates the ambiguity.
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ docker compose -p ordo --profile voice up -d
Hermes config schema does **not** persist a TTS `base_url` (and there is no env for
it), so this Hermes version cannot target local Kokoro for replies. The Kokoro
service is still deployed + registry-managed and reachable at `http://tts:8880/v1`
for n8n / the reel pipeline / scripts / a future Hermes that honours a TTS base URL.
for n8n / the media/render pipeline / scripts / a future Hermes that honours a TTS base URL.
For a fully-local reply voice today, use Hermes' native `neutts` provider (on-device).

**STT** weights download once to `${DATA_PATH}/voice/hf-cache` (persists across
Expand Down
2 changes: 1 addition & 1 deletion docs/operator-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ current split: today there is only Ordo.
4. **Guided-setup wizard** — `ordo setup` detects → proposes → writes `ordo.yaml` (headless path = CI). ✅
5. **Full-stack parity render + `ordo parity`** — the renderer now reproduces the complete llama.cpp surface (model/ctx/mmproj/MTP args/…), and `ordo parity --ref <.env>` diffs it. ✅
**Merge-gate (a) demonstrated live:** `ordo parity` vs the real running `.env` → **PARITY OK** (15 keys, 0 mismatches), read-only — proving the engine regenerates today's hand-tuned config from one source with no drift.
6. **Scheduler status API + `ordo doctor` support bundle** — `Scheduler.status()` emits the busy/idle + free-VRAM + running/queued + ETA JSON the dashboard/agents poll; `ordo doctor [--bundle]` exports a secret-redacted diagnostics bundle. ✅ Demonstrated: a 17GB reel + a 4GB chat **co-run** (chat slips beside the render) — the exact eviction-deadlock that broke primus, gone.
6. **Scheduler status API + `ordo doctor` support bundle** — `Scheduler.status()` emits the busy/idle + free-VRAM + running/queued + ETA JSON the dashboard/agents poll; `ordo doctor [--bundle]` exports a secret-redacted diagnostics bundle. ✅ Demonstrated: a 17GB render job + a 4GB chat **co-run** (chat slips beside the render) — the exact eviction-deadlock that broke primus, gone.
7. **MCP as `kind=mcp` plugins** — an MCP server is a manifest (pinned image + env + tools); the renderer composes enabled ones into `out/mcp-registry.yaml` (drift-free) and flags un-pinned images. Runs on CPU. ✅
8. **Compose rendering** — `ordo render` emits an **isolated, runnable** `docker-compose.yml` (own project/network, no host-port clashes, GPU-gated, profile-gated plugins). ✅ The rendered compose is validated by the **real `docker compose config`** engine (both CPU-core and GPU+media shapes), and that check is a CI gate — not just a well-shaped Python dict.
9. **Process broker** — turns scheduler decisions into real container start/stop; the Docker backend is **hard-scoped to the `ordo-` prefix so it can never touch the live stack**. ✅
Expand Down
2 changes: 1 addition & 1 deletion ordo.example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ agent: hermes
# (the scheduler owns the GPU) and its compose-mutation endpoints are disabled.
dashboard: v2-native

# Optional plugins (media/song/reels/voice). "auto" = enable what the hardware can run;
# Optional plugins (media/song/voice). "auto" = enable what the hardware can run;
# a list forces specific ones. Media plugins are NVIDIA-only and self-declare that.
plugins: auto

Expand Down
4 changes: 2 additions & 2 deletions scripts/storage_purge.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@
its report.

Retention policy (see docs/superpowers/specs/2026-06-19-storage-purge-design.md):
- Reel drafts (data/drafts/<YYYY-MM-DD>_<slug>/): delete > 60 days (analytics
- Job drafts (data/drafts/<YYYY-MM-DD>_<slug>/): delete > 60 days (analytics
window floor). Age comes from the folder-name date prefix, NOT mtime
(reel-metrics rewrites meta.json and would reset mtime).
(metrics rewrites meta.json and would reset mtime).
- ComfyUI output (data/comfyui-output/): delete files > 21 days (mtime).
- ComfyUI input/temp (data/comfyui-storage/ComfyUI/{input,temp}): > 21 days.
- Backups/snapshots (data/_backups/*, data/hermes/state-snapshots/*): keep
Expand Down
2 changes: 1 addition & 1 deletion services/hermes/entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ fi
export HERMES_HOME

# Same ownership/writability check for the gameplay manifest used by the
# social-relay reel pipeline. ComfyUI writes outputs into this directory as
# render output pipeline. ComfyUI writes outputs into this directory as
# root, which leaves the manifest unwritable for the unprivileged hermes user
# the agent's execute_code sandbox runs as (uid 1000). Without this, the cron
# completes its run but cannot record the gameplay segment in `used`, so the
Expand Down
4 changes: 2 additions & 2 deletions services/ops-api/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@
# In the ordo stack the `ordo serve` scheduler (service `ops-controller`) is the SINGLE GPU
# arbiter with full media-lease semantics (evict the resident LLM for a GPU-heavy job, restore it
# on completion, self-heal a stranded lease via TTL). This ops-api must NOT also stop/start
# llama.cpp — two arbiters racing on the same card is exactly the deadlock the reel-cron failure
# llama.cpp — two arbiters racing on the same card is exactly the GPU-arbitration deadlock the failure
# forensics traced. So the guardian's MUTATING routes (/guardian/hold, /guardian/release) are hard-
# disabled and return 410 GONE pointing at the V2 contract. /guardian/status stays live (static
# disabled state) because the dashboard reads it. The scheduler's own broker is the only thing that
Expand Down Expand Up @@ -2174,7 +2174,7 @@ async def guardian_status(_: None = Depends(verify_token)):
async def guardian_hold(body: ConfirmBody, request: Request, _: None = Depends(verify_token)):
"""REMOVED in ordo — 410 GONE. The `ordo serve` scheduler is the single GPU arbiter now;
acquire a media lease via `POST ops-controller:9000/jobs` instead. This route no longer
stops/starts llama.cpp (two arbiters racing on the card was the reel-cron deadlock)."""
stops/starts llama.cpp (two arbiters racing on the card was the GPU-arbitration deadlock)."""
_audit("guardian_hold", COMFYUI_GUARDIAN_TARGET, "gone", "410 -> ops-controller /jobs",
correlation_id=_correlation_id(request))
raise HTTPException(status_code=410, detail=_GUARDIAN_MUTATION_GONE_DETAIL)
Expand Down
2 changes: 1 addition & 1 deletion services/worker/plugin.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Plugin manifest (data-only) — the background job worker (reel/publish queue). LOCALLY-BUILT in V1
# Plugin manifest (data-only) — the background job worker (job/publish queue). LOCALLY-BUILT in V1
# (dashboard + worker deps); V2 pins it as a project buildable image. Drives ComfyUI render jobs and
# publish enqueues, so it depends on the comfyui plugin (and the core dashboard for the job store).
id: worker
Expand Down
28 changes: 14 additions & 14 deletions tests/substrate/test_broker.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,44 +66,44 @@ def _broker_with_resident(total=32, resident_gb=25):

def test_media_lease_stops_resident_then_restarts_it_on_completion():
b = _broker_with_resident()
b.request(Job("reel", 18, "media")) # evicts llamacpp -> stop; starts reel
b.request(Job("render", 18, "media")) # evicts llamacpp -> stop; starts render
assert "llamacpp" in b.backend.stopped
assert "reel" in b.backend.started
assert "render" in b.backend.started
b.backend.started.clear() # focus on what completion does
b.complete("reel") # media done -> restore the resident
assert "reel" in b.backend.stopped # the job container is stopped
b.complete("render") # media done -> restore the resident
assert "render" in b.backend.stopped # the job container is stopped
assert b.backend.started == ["llamacpp"] # resident RESTARTED by the broker


def test_no_resident_restart_between_back_to_back_renders():
b = _broker_with_resident()
b.request(Job("reel1", 18, "media")) # reel1 runs (llamacpp stopped)
b.request(Job("reel2", 18, "media")) # reel2 queued
b.request(Job("render1", 18, "media")) # render1 runs (llamacpp stopped)
b.request(Job("render2", 18, "media")) # render2 queued
b.backend.started.clear()
b.complete("reel1") # reel2 admitted — resident NOT restarted
assert "reel2" in b.backend.started
assert "llamacpp" not in b.backend.started # anti-thrash: still down for reel2
b.complete("render1") # render2 admitted — resident NOT restarted
assert "render2" in b.backend.started
assert "llamacpp" not in b.backend.started # anti-thrash: still down for render2
b.backend.started.clear()
b.complete("reel2")
b.complete("render2")
assert b.backend.started == ["llamacpp"] # only after the queue fully drains


def test_sweep_leases_restarts_resident_for_a_stranded_job():
s = Scheduler(32)
s.cache_idle("llamacpp", 25)
b = Broker(s, MockBackend())
b.request(Job("reel", 18, "media", est_seconds=30)) # TTL = 60s
b.request(Job("render", 18, "media", est_seconds=30)) # TTL = 60s
b.backend.started.clear()
s.tick(70) # past the 60s TTL (serve loop ticks the clock)
swept = b.sweep_leases()
assert swept == ["reel"]
assert "reel" in b.backend.stopped # stranded job's container stopped
assert swept == ["render"]
assert "render" in b.backend.stopped # stranded job's container stopped
assert b.backend.started == ["llamacpp"] # resident self-healed back up


def test_sweep_is_noop_when_no_lease_expired():
b = _broker_with_resident()
b.request(Job("reel", 18, "media", est_seconds=600))
b.request(Job("render", 18, "media", est_seconds=600))
b.backend.started.clear()
b.backend.stopped.clear()
b.scheduler.tick(5)
Expand Down
18 changes: 9 additions & 9 deletions tests/substrate/test_control.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,12 +64,12 @@ def test_set_unknown_model_is_404_and_writes_nothing(tmp_path):

def test_job_lifecycle_drives_scheduler(tmp_path):
cp, _ = _cp(tmp_path)
cp.route("POST", "/jobs", {"id": "reel", "vram_gb": 17, "kind": "media"})
cp.route("POST", "/jobs", {"id": "render", "vram_gb": 17, "kind": "media"})
cp.route("POST", "/jobs", {"id": "chat", "vram_gb": 4, "kind": "chat"})
st = cp.route("GET", "/status")[1]["gpu"]
running = {r["id"] for r in st["running"]}
assert running == {"reel", "chat"} # co-run: chat slips beside the reel
_, after = cp.route("POST", "/jobs/complete", {"id": "reel"})
assert running == {"render", "chat"} # co-run: chat slips beside the render
_, after = cp.route("POST", "/jobs/complete", {"id": "render"})
assert {r["id"] for r in after["running"]} == {"chat"}


Expand All @@ -96,7 +96,7 @@ def _cp_with_resident(tmp_path, resident_gb=25):
def test_status_exposes_evicted_residents_and_lease_ttl(tmp_path):
cp, _ = _cp_with_resident(tmp_path)
# a media job that doesn't fit beside the resident -> evicts + stops it via the broker
code, body = cp.route("POST", "/jobs", {"id": "reel", "vram_gb": 18, "kind": "media",
code, body = cp.route("POST", "/jobs", {"id": "render", "vram_gb": 18, "kind": "media",
"est_seconds": 120})
assert code == 200
assert body["evicted_residents"] == {"llamacpp": 25.0} # lease surface in the /jobs response
Expand All @@ -108,9 +108,9 @@ def test_status_exposes_evicted_residents_and_lease_ttl(tmp_path):

def test_complete_job_restores_resident_over_control_plane(tmp_path):
cp, _ = _cp_with_resident(tmp_path)
cp.route("POST", "/jobs", {"id": "reel", "vram_gb": 18, "kind": "media"})
cp.route("POST", "/jobs", {"id": "render", "vram_gb": 18, "kind": "media"})
assert cp.scheduler.evicted_residents == {"llamacpp": 25}
_, after = cp.route("POST", "/jobs/complete", {"id": "reel"})
_, after = cp.route("POST", "/jobs/complete", {"id": "render"})
# completing the media job drains the queue -> broker restores the resident; status reflects it
assert after["evicted_residents"] == {}
assert after["idle_cached"] == {"llamacpp": 25.0} # re-armed as evictable
Expand Down Expand Up @@ -148,11 +148,11 @@ def test_jobs_history_route_serves_lease_records(tmp_path):
cp, _ = _cp(tmp_path)
cp.broker.history = hist
cp.history = hist
cp.route("POST", "/jobs", {"id": "reel", "vram_gb": 17, "kind": "media"})
cp.route("POST", "/jobs/complete", {"id": "reel"})
cp.route("POST", "/jobs", {"id": "render", "vram_gb": 17, "kind": "media"})
cp.route("POST", "/jobs/complete", {"id": "render"})
code, body = cp.route("GET", "/jobs/history")
assert code == 200
assert body["history"][0]["id"] == "reel"
assert body["history"][0]["id"] == "render"
assert body["history"][0]["outcome"] == "completed"


Expand Down
Loading
Loading