Predictive Maintenance Demo

Agentium predictive-maintenance copilot over a simulated CVD chamber: continuous SCADA telemetry, MES logs, Grafana alerts, fault injection, investigator agents, maintenance report, Slack notify.

Operator detail: OPERATING.md

Prereqs

• docker, kubectl, helm, k3d, jq, curl, just
• Demo image agentium-runner:demo — the single agentium binary (runner + builder + SDK) running agentium serve; loaded into k3d. Build with just build-release then docker build -t agentium-runner:demo . if missing.
• .env with secrets (see below)

One binary. Agentium now ships a single agentium binary that wraps the former baml-agent-runner (runner → agentium serve), the build pipeline (server-side build on publish), and cargo-agent-platform (SDK subcommands: install, deploy, chat, eval, new-agent). There is no separate runner, builder, or cargo-agent-platform binary. See docs/reference/agentium-cli.md.

Configure

cp .env_example .env
$EDITOR .env

Minimum for a full run:

OPENROUTER_API_KEY=...
GRAFANA_PASSWORD=admin
SLACK_BOT_TOKEN=xoxb-...
SLACK_NOTIFY_CHANNEL_ID=C0123456789   # channel ID, not name

Optional: RUNNER_TOKEN=demo-runner-token (chart default; needed only for host-side deploy-agents).

Run demo

One command installs cluster stack, service images, Helm chart, Grafana MCP, and all three agents via in-cluster deployer Job.

just nuke k3d
just pf-start

just pf-start runs port-forwards in background. Alternative: just port-forward blocks current terminal and stops forwards on Ctrl-C.

Stop any local runner on :18080 before port-forwarding.

Verify:

kubectl -n agentium-demo get pods
curl -s http://127.0.0.1:18080/agents | jq -r '.[].agent_package'
# maintenance-coordinator, telemetry-investigator, slack-notify

Open Grafana alert query view, keep port-forwards running, wait ≥10 minutes for baseline telemetry, then:

just inject

Reset / smoke test (cluster must already be installed — just install or just nuke first):

just reset
just e2e

Fresh install + e2e in one shot:

SKIP_INSTALL=0 just e2e

URLs (just pf-start)

Service	URL
Grafana	http://127.0.0.1:3000 (admin / $GRAFANA_PASSWORD)
HighChamberTemperature alert	http://127.0.0.1:3000/alerting/list
Agentium runner	http://127.0.0.1:18080
Agentium dashboard	http://127.0.0.1:18080/?view=dashboard&contextId=<id>

What just nuke deploys

In cluster	Purpose
stepper-sim	Simulated CVD chamber: temperature, vibration, pressure, coolant flow, MES logs
fault-injector	Seeded bounded fault injection, Grafana CMMS/window annotations, ground-truth ledger
grafana, prometheus, loki, alloy	Observability stack
agentium-runner	Single agentium binary via agentium serve (runner + server-side builder + SDK) + MCP runtime
agentium-demo-agent-deployer (Job)	Enables Grafana MCP, publishes agents (server-side build on publish)

Grafana MCP runs in the cluster agentium image (/usr/local/bin/mcp-grafana) and Helm ConfigMap agentium-demo-mcp.

Common commands

Goal	Command
Build the agentium binary	just build-release (produces target/release/agentium)
Fresh install	just nuke k3d
Install only	just setup k3d or just install
First-time cluster + install	just up
Port-forwards	just pf-start / just pf-stop / just pf-status
Workspace overview	just status
Follow runner logs	just runner-logs
Rebuild images + restart	just reload k3d
Rebuild runner from a branch	see Rebuilding the runner from agentium-os source
Inject fault	just inject
Push agent edits from laptop	just deploy-agents

Equivalent shell entrypoint: ./demo.sh install|inject|reset|e2e.

Rebuilding the runner from agentium-os source

By default just build-runner builds in release mode — it pulls the prebuilt agentium-os binary pinned by AGENTIUM_OS_VERSION (default v0.1.2). To run a specific agentium-os branch or commit instead, use source mode, which clones and compiles AGENTIUM_OS_REF inside the image build.

Env var	Default	Effect
RUNNER_BUILD_MODE	release	source clones + compiles agentium-os instead of using a released binary
AGENTIUM_OS_REF	main	branch/tag to build (source mode only)
AGENTIUM_OS_REPO	…/agentium-os.git	repo to clone (source mode only)
AGENTIUM_OS_VERSION	v0.1.2	released binary version (release mode only)
BUILD_RUNNER_IMAGE	0	1 makes build-images / reload actually build the runner
DOCKER_NO_CACHE	0	1 forces a fresh clone + recompile (source mode only)

Why DOCKER_NO_CACHE: Docker caches the git clone layer on the command string, not the branch HEAD — so without it a rebuild silently reuses whatever commit was cloned first, and new pushes to the branch are invisible. Set DOCKER_NO_CACHE=1 to re-clone and recompile from the current branch HEAD (base/apt layers stay cached).

Example — build + deploy the runner from main

Build the image only (verify the branch compiles; no cluster changes):

DOCKER_NO_CACHE=1 RUNNER_BUILD_MODE=source AGENTIUM_OS_REF=main \
just build-runner

Build from main, import into the running cluster, and restart the runner + services in one step:

DOCKER_NO_CACHE=1 BUILD_RUNNER_IMAGE=1 RUNNER_BUILD_MODE=source \
AGENTIUM_OS_REF=main just reload

just reload re-clones + recompiles, imports the image into the current agentium cluster, restarts agentium-runner + stepper-sim + fault-injector, and repacks the agents — no cluster recreate. Plain just up / just reload stay cached and fast for the common case (no runner change). Swap AGENTIUM_OS_REF=main for any branch (e.g. a feature branch) to test it.

The cargo compile takes several minutes and pegs CPU with rustc — that's normal, not a hang. Watch from another shell:

watch -n5 'ps -eo etime,pcpu,comm | grep -E "rustc|cargo"'

Local agent iteration

just pf-start
RUNNER_TOKEN=demo-runner-token just deploy-agents

Skip MCP setup if cluster already enabled it: SKIP_MCP_ENABLE=1 just deploy-agents.

After agent-only changes without local push: just pack-agents && just install.

Honest framing

• Metrics and MES logs are real Prometheus/Loki streams from live pods.
• CMMS/service history is synthetic Grafana annotations from fault-injector.
• Agent never sees fault-injector ground truth ledger; grader joins ledger with provenance later.
• Final report lives in Agentium provenance; Slack gets summary pointer.

Troubleshooting

Symptom	Check
Empty /agents on :18080	Local runner blocking port-forward; use cluster runner via just pf-start
deploy-agents token error	Set RUNNER_TOKEN=demo-runner-token in .env
grafana not in MCP config	Normal for host deploy; skip with SKIP_MCP_ENABLE=1
No alert after inject	Wait for warm-up; check scada_* metrics in Prometheus; alert eval may lag ~1 min
Hook Job failed	kubectl -n agentium-demo logs -l job-name=agentium-demo-agent-deployer --all-containers

More: OPERATING.md, helm/README.md.