diff --git a/CHANGELOG.md b/CHANGELOG.md index 4ae5b2a4..47b2c0a7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -87,7 +87,7 @@ All notable changes to this project are documented here. The format is loosely b ### Fixed - **GitHub-monitor cron no longer blocked by invisible unicode.** `scripts/stack_monitor.py` fetches upstream GitHub release names and commit messages, which routinely embed zero-width / invisible "format" (Cf) characters β€” most often the ZWJ `U+200D` inside emoji sequences like πŸ‘¨β€πŸ’». When the JSON report was fed back into Hermes to format for Discord, that invisible unicode tripped the prompt-injection scanner and the whole daily cron failed (`Blocked: prompt contains invisible unicode U+200D`), every run. The monitor now strips all Cf-category characters from the entire output (recursively, both `--json` and human-readable) before emitting; visible text and emoji are unaffected (a ZWJ emoji renders as its component glyphs). Covered by `tests/test_stack_monitor_sanitize.py`. -- **ComfyUI restart storm hardened (status verb + server-side debounce).** During a failed reel cron, Hermes tried to recover ComfyUI by hammering guessed paths `POST /api/comfyui/restart` and `GET /api/comfyui/status` against both ops-controller (404) and the dashboard (401), ~every 3s for ~2 min β€” it had no canonical *status* verb so it improvised raw HTTP. Two fixes: (1) ops-controller `POST /services/{id}/restart` now **debounces** rapid repeats per-service (`OPS_RESTART_DEBOUNCE_SECONDS`, default 20) so a retry-loop collapses into one in-flight restart instead of stacking `docker restart` calls; (2) a new ComfyUI-**independent** health verb β€” dashboard `GET /api/orchestration/comfyui/status` + orchestration-MCP `comfyui_status` tool β€” that reports container state + render-queue reachability by querying the dashboardβ†’ops control plane (which stays up when ComfyUI is down), so the agent checks status via a real tool instead of guessing. The `restart_comfyui` tool docstring now points agents at `comfyui_status` rather than tight-looping. +- **ComfyUI restart storm hardened (status verb + server-side debounce).** During a failed render cron, Hermes tried to recover ComfyUI by hammering guessed paths `POST /api/comfyui/restart` and `GET /api/comfyui/status` against both ops-controller (404) and the dashboard (401), ~every 3s for ~2 min β€” it had no canonical *status* verb so it improvised raw HTTP. Two fixes: (1) ops-controller `POST /services/{id}/restart` now **debounces** rapid repeats per-service (`OPS_RESTART_DEBOUNCE_SECONDS`, default 20) so a retry-loop collapses into one in-flight restart instead of stacking `docker restart` calls; (2) a new ComfyUI-**independent** health verb β€” dashboard `GET /api/orchestration/comfyui/status` + orchestration-MCP `comfyui_status` tool β€” that reports container state + render-queue reachability by querying the dashboardβ†’ops control plane (which stays up when ComfyUI is down), so the agent checks status via a real tool instead of guessing. The `restart_comfyui` tool docstring now points agents at `comfyui_status` rather than tight-looping. - **Dashboard at `/dash/` renders fully behind SSO front door (#14).** Three coupled bugs prevented the dashboard from working when served at `/dash/`: (a) all `fetch('/api/...')` calls used absolute paths and missed Caddy's `/dash/*` prefix β†’ `index.html` now detects `APP_PREFIX` from `location.pathname` and prepends it inside the `api()` wrapper, the bootstrap `/api/auth/config` fetch, and the favicon link; (b) service-card links pointed at direct upstream ports (e.g. `:8188`) that aren't host-published β€” `serviceOpenHref` now maps known SSO-gated services (`comfyui` β†’ `/comfy/`, `webui` β†’ `/chat/`, `n8n` β†’ `/n8n/`, `hermes` β†’ `/hermes/`) to their Caddy paths when `APP_PREFIX` is set; (c) `/api/auth/config` advertised `auth_type=bearer` even when the request arrived from the trusted reverse proxy with a verified `X-Forwarded-Email` β†’ endpoint now returns `auth_required=false` on the SSO path so the bearer-token modal doesn't pop up. - **Caddy `route { forward_auth + handle_path }` pattern restored on main.** The original `handle @auth { forward_auth }` shape was terminal β€” once `forward_auth` returned 202, the request ended without ever reaching the path-prefix handler, so SSO-gated routes (`/dash/`, `/chat/`, `/n8n/`, `/hermes/`, `/comfy/`) returned an empty 202 instead of the upstream UI. Pairs with the `--email-domain=*` removal in oauth2-proxy (the wildcard bypassed the `--authenticated-emails-file` allowlist). - **Caddy moved off `frontend` network.** Membership on both `proxy-net` and `frontend` made Docker DNS return whichever IP it picked first for upstream services. When it chose the `frontend` IP, dashboard saw caddy as outside `DASHBOARD_TRUSTED_PROXY_NET=172.24.0.0/16`, refused the `X-Forwarded-Email` header, and 401'd every `/api/*` call. Caddy is the SSO ingress only and reaches every upstream over `proxy-net`; dropping `frontend` membership eliminates the ambiguity. diff --git a/docs/configuration.md b/docs/configuration.md index f76f0f35..b91c1329 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -93,7 +93,7 @@ docker compose -p ordo --profile voice up -d Hermes config schema does **not** persist a TTS `base_url` (and there is no env for it), so this Hermes version cannot target local Kokoro for replies. The Kokoro service is still deployed + registry-managed and reachable at `http://tts:8880/v1` - for n8n / the reel pipeline / scripts / a future Hermes that honours a TTS base URL. + for n8n / the media/render pipeline / scripts / a future Hermes that honours a TTS base URL. For a fully-local reply voice today, use Hermes' native `neutts` provider (on-device). **STT** weights download once to `${DATA_PATH}/voice/hf-cache` (persists across diff --git a/docs/operator-guide.md b/docs/operator-guide.md index 5bb7aba5..476fc005 100644 --- a/docs/operator-guide.md +++ b/docs/operator-guide.md @@ -52,7 +52,7 @@ current split: today there is only Ordo. 4. **Guided-setup wizard** β€” `ordo setup` detects β†’ proposes β†’ writes `ordo.yaml` (headless path = CI). βœ… 5. **Full-stack parity render + `ordo parity`** β€” the renderer now reproduces the complete llama.cpp surface (model/ctx/mmproj/MTP args/…), and `ordo parity --ref <.env>` diffs it. βœ… **Merge-gate (a) demonstrated live:** `ordo parity` vs the real running `.env` β†’ **PARITY OK** (15 keys, 0 mismatches), read-only β€” proving the engine regenerates today's hand-tuned config from one source with no drift. -6. **Scheduler status API + `ordo doctor` support bundle** β€” `Scheduler.status()` emits the busy/idle + free-VRAM + running/queued + ETA JSON the dashboard/agents poll; `ordo doctor [--bundle]` exports a secret-redacted diagnostics bundle. βœ… Demonstrated: a 17GB reel + a 4GB chat **co-run** (chat slips beside the render) β€” the exact eviction-deadlock that broke primus, gone. +6. **Scheduler status API + `ordo doctor` support bundle** β€” `Scheduler.status()` emits the busy/idle + free-VRAM + running/queued + ETA JSON the dashboard/agents poll; `ordo doctor [--bundle]` exports a secret-redacted diagnostics bundle. βœ… Demonstrated: a 17GB render job + a 4GB chat **co-run** (chat slips beside the render) β€” the exact eviction-deadlock that broke primus, gone. 7. **MCP as `kind=mcp` plugins** β€” an MCP server is a manifest (pinned image + env + tools); the renderer composes enabled ones into `out/mcp-registry.yaml` (drift-free) and flags un-pinned images. Runs on CPU. βœ… 8. **Compose rendering** β€” `ordo render` emits an **isolated, runnable** `docker-compose.yml` (own project/network, no host-port clashes, GPU-gated, profile-gated plugins). βœ… The rendered compose is validated by the **real `docker compose config`** engine (both CPU-core and GPU+media shapes), and that check is a CI gate β€” not just a well-shaped Python dict. 9. **Process broker** β€” turns scheduler decisions into real container start/stop; the Docker backend is **hard-scoped to the `ordo-` prefix so it can never touch the live stack**. βœ… diff --git a/ordo.example.yaml b/ordo.example.yaml index fb8cf744..1ef99cbc 100644 --- a/ordo.example.yaml +++ b/ordo.example.yaml @@ -29,7 +29,7 @@ agent: hermes # (the scheduler owns the GPU) and its compose-mutation endpoints are disabled. dashboard: v2-native -# Optional plugins (media/song/reels/voice). "auto" = enable what the hardware can run; +# Optional plugins (media/song/voice). "auto" = enable what the hardware can run; # a list forces specific ones. Media plugins are NVIDIA-only and self-declare that. plugins: auto diff --git a/scripts/storage_purge.py b/scripts/storage_purge.py index e92c7bf2..6e83efe0 100644 --- a/scripts/storage_purge.py +++ b/scripts/storage_purge.py @@ -7,9 +7,9 @@ its report. Retention policy (see docs/superpowers/specs/2026-06-19-storage-purge-design.md): - - Reel drafts (data/drafts/_/): delete > 60 days (analytics + - Job drafts (data/drafts/_/): delete > 60 days (analytics window floor). Age comes from the folder-name date prefix, NOT mtime - (reel-metrics rewrites meta.json and would reset mtime). + (metrics rewrites meta.json and would reset mtime). - ComfyUI output (data/comfyui-output/): delete files > 21 days (mtime). - ComfyUI input/temp (data/comfyui-storage/ComfyUI/{input,temp}): > 21 days. - Backups/snapshots (data/_backups/*, data/hermes/state-snapshots/*): keep diff --git a/services/hermes/entrypoint.sh b/services/hermes/entrypoint.sh index f62c1ce1..47380395 100644 --- a/services/hermes/entrypoint.sh +++ b/services/hermes/entrypoint.sh @@ -22,7 +22,7 @@ fi export HERMES_HOME # Same ownership/writability check for the gameplay manifest used by the -# social-relay reel pipeline. ComfyUI writes outputs into this directory as +# render output pipeline. ComfyUI writes outputs into this directory as # root, which leaves the manifest unwritable for the unprivileged hermes user # the agent's execute_code sandbox runs as (uid 1000). Without this, the cron # completes its run but cannot record the gameplay segment in `used`, so the diff --git a/services/ops-api/main.py b/services/ops-api/main.py index f59f0a22..ada24209 100644 --- a/services/ops-api/main.py +++ b/services/ops-api/main.py @@ -112,7 +112,7 @@ # In the ordo stack the `ordo serve` scheduler (service `ops-controller`) is the SINGLE GPU # arbiter with full media-lease semantics (evict the resident LLM for a GPU-heavy job, restore it # on completion, self-heal a stranded lease via TTL). This ops-api must NOT also stop/start -# llama.cpp β€” two arbiters racing on the same card is exactly the deadlock the reel-cron failure +# llama.cpp β€” two arbiters racing on the same card is exactly the GPU-arbitration deadlock the failure # forensics traced. So the guardian's MUTATING routes (/guardian/hold, /guardian/release) are hard- # disabled and return 410 GONE pointing at the V2 contract. /guardian/status stays live (static # disabled state) because the dashboard reads it. The scheduler's own broker is the only thing that @@ -2174,7 +2174,7 @@ async def guardian_status(_: None = Depends(verify_token)): async def guardian_hold(body: ConfirmBody, request: Request, _: None = Depends(verify_token)): """REMOVED in ordo β€” 410 GONE. The `ordo serve` scheduler is the single GPU arbiter now; acquire a media lease via `POST ops-controller:9000/jobs` instead. This route no longer - stops/starts llama.cpp (two arbiters racing on the card was the reel-cron deadlock).""" + stops/starts llama.cpp (two arbiters racing on the card was the GPU-arbitration deadlock).""" _audit("guardian_hold", COMFYUI_GUARDIAN_TARGET, "gone", "410 -> ops-controller /jobs", correlation_id=_correlation_id(request)) raise HTTPException(status_code=410, detail=_GUARDIAN_MUTATION_GONE_DETAIL) diff --git a/services/worker/plugin.yaml b/services/worker/plugin.yaml index f1a643fd..8131dc56 100644 --- a/services/worker/plugin.yaml +++ b/services/worker/plugin.yaml @@ -1,4 +1,4 @@ -# Plugin manifest (data-only) β€” the background job worker (reel/publish queue). LOCALLY-BUILT in V1 +# Plugin manifest (data-only) β€” the background job worker (job/publish queue). LOCALLY-BUILT in V1 # (dashboard + worker deps); V2 pins it as a project buildable image. Drives ComfyUI render jobs and # publish enqueues, so it depends on the comfyui plugin (and the core dashboard for the job store). id: worker diff --git a/tests/substrate/test_broker.py b/tests/substrate/test_broker.py index 2063fea1..1be74a05 100644 --- a/tests/substrate/test_broker.py +++ b/tests/substrate/test_broker.py @@ -66,25 +66,25 @@ def _broker_with_resident(total=32, resident_gb=25): def test_media_lease_stops_resident_then_restarts_it_on_completion(): b = _broker_with_resident() - b.request(Job("reel", 18, "media")) # evicts llamacpp -> stop; starts reel + b.request(Job("render", 18, "media")) # evicts llamacpp -> stop; starts render assert "llamacpp" in b.backend.stopped - assert "reel" in b.backend.started + assert "render" in b.backend.started b.backend.started.clear() # focus on what completion does - b.complete("reel") # media done -> restore the resident - assert "reel" in b.backend.stopped # the job container is stopped + b.complete("render") # media done -> restore the resident + assert "render" in b.backend.stopped # the job container is stopped assert b.backend.started == ["llamacpp"] # resident RESTARTED by the broker def test_no_resident_restart_between_back_to_back_renders(): b = _broker_with_resident() - b.request(Job("reel1", 18, "media")) # reel1 runs (llamacpp stopped) - b.request(Job("reel2", 18, "media")) # reel2 queued + b.request(Job("render1", 18, "media")) # render1 runs (llamacpp stopped) + b.request(Job("render2", 18, "media")) # render2 queued b.backend.started.clear() - b.complete("reel1") # reel2 admitted β€” resident NOT restarted - assert "reel2" in b.backend.started - assert "llamacpp" not in b.backend.started # anti-thrash: still down for reel2 + b.complete("render1") # render2 admitted β€” resident NOT restarted + assert "render2" in b.backend.started + assert "llamacpp" not in b.backend.started # anti-thrash: still down for render2 b.backend.started.clear() - b.complete("reel2") + b.complete("render2") assert b.backend.started == ["llamacpp"] # only after the queue fully drains @@ -92,18 +92,18 @@ def test_sweep_leases_restarts_resident_for_a_stranded_job(): s = Scheduler(32) s.cache_idle("llamacpp", 25) b = Broker(s, MockBackend()) - b.request(Job("reel", 18, "media", est_seconds=30)) # TTL = 60s + b.request(Job("render", 18, "media", est_seconds=30)) # TTL = 60s b.backend.started.clear() s.tick(70) # past the 60s TTL (serve loop ticks the clock) swept = b.sweep_leases() - assert swept == ["reel"] - assert "reel" in b.backend.stopped # stranded job's container stopped + assert swept == ["render"] + assert "render" in b.backend.stopped # stranded job's container stopped assert b.backend.started == ["llamacpp"] # resident self-healed back up def test_sweep_is_noop_when_no_lease_expired(): b = _broker_with_resident() - b.request(Job("reel", 18, "media", est_seconds=600)) + b.request(Job("render", 18, "media", est_seconds=600)) b.backend.started.clear() b.backend.stopped.clear() b.scheduler.tick(5) diff --git a/tests/substrate/test_control.py b/tests/substrate/test_control.py index 7b55760a..832a1449 100644 --- a/tests/substrate/test_control.py +++ b/tests/substrate/test_control.py @@ -64,12 +64,12 @@ def test_set_unknown_model_is_404_and_writes_nothing(tmp_path): def test_job_lifecycle_drives_scheduler(tmp_path): cp, _ = _cp(tmp_path) - cp.route("POST", "/jobs", {"id": "reel", "vram_gb": 17, "kind": "media"}) + cp.route("POST", "/jobs", {"id": "render", "vram_gb": 17, "kind": "media"}) cp.route("POST", "/jobs", {"id": "chat", "vram_gb": 4, "kind": "chat"}) st = cp.route("GET", "/status")[1]["gpu"] running = {r["id"] for r in st["running"]} - assert running == {"reel", "chat"} # co-run: chat slips beside the reel - _, after = cp.route("POST", "/jobs/complete", {"id": "reel"}) + assert running == {"render", "chat"} # co-run: chat slips beside the render + _, after = cp.route("POST", "/jobs/complete", {"id": "render"}) assert {r["id"] for r in after["running"]} == {"chat"} @@ -96,7 +96,7 @@ def _cp_with_resident(tmp_path, resident_gb=25): def test_status_exposes_evicted_residents_and_lease_ttl(tmp_path): cp, _ = _cp_with_resident(tmp_path) # a media job that doesn't fit beside the resident -> evicts + stops it via the broker - code, body = cp.route("POST", "/jobs", {"id": "reel", "vram_gb": 18, "kind": "media", + code, body = cp.route("POST", "/jobs", {"id": "render", "vram_gb": 18, "kind": "media", "est_seconds": 120}) assert code == 200 assert body["evicted_residents"] == {"llamacpp": 25.0} # lease surface in the /jobs response @@ -108,9 +108,9 @@ def test_status_exposes_evicted_residents_and_lease_ttl(tmp_path): def test_complete_job_restores_resident_over_control_plane(tmp_path): cp, _ = _cp_with_resident(tmp_path) - cp.route("POST", "/jobs", {"id": "reel", "vram_gb": 18, "kind": "media"}) + cp.route("POST", "/jobs", {"id": "render", "vram_gb": 18, "kind": "media"}) assert cp.scheduler.evicted_residents == {"llamacpp": 25} - _, after = cp.route("POST", "/jobs/complete", {"id": "reel"}) + _, after = cp.route("POST", "/jobs/complete", {"id": "render"}) # completing the media job drains the queue -> broker restores the resident; status reflects it assert after["evicted_residents"] == {} assert after["idle_cached"] == {"llamacpp": 25.0} # re-armed as evictable @@ -148,11 +148,11 @@ def test_jobs_history_route_serves_lease_records(tmp_path): cp, _ = _cp(tmp_path) cp.broker.history = hist cp.history = hist - cp.route("POST", "/jobs", {"id": "reel", "vram_gb": 17, "kind": "media"}) - cp.route("POST", "/jobs/complete", {"id": "reel"}) + cp.route("POST", "/jobs", {"id": "render", "vram_gb": 17, "kind": "media"}) + cp.route("POST", "/jobs/complete", {"id": "render"}) code, body = cp.route("GET", "/jobs/history") assert code == 200 - assert body["history"][0]["id"] == "reel" + assert body["history"][0]["id"] == "render" assert body["history"][0]["outcome"] == "completed" diff --git a/tests/substrate/test_scheduler.py b/tests/substrate/test_scheduler.py index 09b2dd1c..0d9c584a 100644 --- a/tests/substrate/test_scheduler.py +++ b/tests/substrate/test_scheduler.py @@ -84,9 +84,9 @@ def _sched_with_resident(total=32, resident_gb=25): def test_media_job_evicts_resident_and_records_it_for_restore(): s = _sched_with_resident() # 32 total, 25 resident -> 7 free (a media job won't fit) - s.submit(Job("reel", 18, "media")) + s.submit(Job("render", 18, "media")) admitted, evicted = s.pump() - assert admitted == ["reel"] + assert admitted == ["render"] assert evicted == ["llamacpp"] # resident stopped to free VRAM assert "llamacpp" in s.evicted_residents # tracked with its footprint for restore assert s.evicted_residents["llamacpp"] == 25 @@ -95,9 +95,9 @@ def test_media_job_evicts_resident_and_records_it_for_restore(): def test_resident_restored_when_media_completes_and_queue_drains(): s = _sched_with_resident() - s.submit(Job("reel", 18, "media")) - s.pump() # evicts llamacpp, runs reel - s.complete("reel") # media done -> queue empty + s.submit(Job("render", 18, "media")) + s.pump() # evicts llamacpp, runs render + s.complete("render") # media done -> queue empty restored = s.take_restorable() assert restored == {"llamacpp": 25} # resident restarted with its footprint assert "llamacpp" in s.idle_cached # re-armed as evictable for the next lease @@ -106,9 +106,9 @@ def test_resident_restored_when_media_completes_and_queue_drains(): def test_no_restore_while_a_media_job_still_running(): s = _sched_with_resident() - s.submit(Job("reel", 18, "media")) + s.submit(Job("render", 18, "media")) s.pump() - # reel still running β€” restoring the LLM now would immediately need re-eviction: don't. + # render still running β€” restoring the LLM now would immediately need re-eviction: don't. assert s.take_restorable() == {} assert "llamacpp" in s.evicted_residents @@ -117,16 +117,16 @@ def test_no_thrash_between_back_to_back_media_jobs(): # Two queued media renders: complete the first, and the resident must NOT flap on/off β€” it stays # evicted until the WHOLE media queue drains (the anti-thrash rule). s = _sched_with_resident() - s.submit(Job("reel1", 18, "media")) - s.submit(Job("reel2", 18, "media")) - s.pump() # reel1 runs (evicts llamacpp); reel2 queued - assert s.running_ids == ["reel1"] and s.queued_ids == ["reel2"] - s.complete("reel1") # reel2 now admittable - assert s.take_restorable() == {} # DON'T restore β€” reel2 still needs the card - s.pump() # reel2 admitted - assert s.running_ids == ["reel2"] - assert s.take_restorable() == {} # still no restore while reel2 runs - s.complete("reel2") + s.submit(Job("render1", 18, "media")) + s.submit(Job("render2", 18, "media")) + s.pump() # render1 runs (evicts llamacpp); render2 queued + assert s.running_ids == ["render1"] and s.queued_ids == ["render2"] + s.complete("render1") # render2 now admittable + assert s.take_restorable() == {} # DON'T restore β€” render2 still needs the card + s.pump() # render2 admitted + assert s.running_ids == ["render2"] + assert s.take_restorable() == {} # still no restore while render2 runs + s.complete("render2") assert s.take_restorable() == {"llamacpp": 25} # only now β€” queue fully drained @@ -134,13 +134,13 @@ def test_lease_ttl_auto_completes_stranded_job(): # A crashed client never calls complete() β€” the TTL sweep force-completes the lease so the # resident can be restored (self-heal; a stranded lease must never kill the brain forever). s = _sched_with_resident() - s.submit(Job("reel", 18, "media", est_seconds=60)) # TTL = 60 * 2 = 120s + s.submit(Job("render", 18, "media", est_seconds=60)) # TTL = 60 * 2 = 120s s.pump() s.tick(90) assert s.sweep_expired_leases() == [] # not yet past the 120s TTL - assert s.running_ids == ["reel"] + assert s.running_ids == ["render"] s.tick(40) # now 130s elapsed > 120s TTL - assert s.sweep_expired_leases() == ["reel"] # force-completed + assert s.sweep_expired_leases() == ["render"] # force-completed assert s.running_ids == [] assert s.take_restorable() == {"llamacpp": 25} # resident restored after the stranded lease @@ -163,7 +163,7 @@ def test_recache_resident_clears_evicted_and_rearms(): # take_restorable re-caches; an explicit cache_idle (e.g. after a manual restart) must also clear # the evicted record so the resident is a normal evictable again. s = _sched_with_resident() - s.submit(Job("reel", 18, "media")) + s.submit(Job("render", 18, "media")) s.pump() assert "llamacpp" in s.evicted_residents s.cache_idle("llamacpp", 25) # re-registered as resident @@ -173,7 +173,7 @@ def test_recache_resident_clears_evicted_and_rearms(): def test_status_surfaces_lease_and_resident_fields(): s = _sched_with_resident() - s.submit(Job("reel", 18, "media", est_seconds=120)) + s.submit(Job("render", 18, "media", est_seconds=120)) s.pump() st = s.status() assert st["evicted_residents"] == {"llamacpp": 25}