QSense Hub — Factory AI on Snapdragon

Part of QSense Factory · Snapdragon Multiverse Hackathon

What is QSense Factory?

QSense Factory is a privacy-first, on-premises factory-safety system for MSMEs. No cloud, no subscription, no video or vibration data ever leaves the floor — everything runs on hardware already sitting in the plant.

Device	Role	Stage
Arduino UNO Q (QSense-Node)	On-board vibration anomaly model per machine, publishes a live severity score	Sense
Snapdragon X Elite Copilot+ PC (this repo)	NPU-accelerated PPE compliance vision, MQTT alert brain, SQLite history, admin dashboard	Think
Mobile (any phone on the LAN)	Same responsive dashboard — supervisors acknowledge / resolve alerts from the floor	Act

The three devices form a closed loop over a local Mosquitto broker running on the hub PC: nodes stream machine severity in, the hub turns thresholds into alerts and pushes them live to every open dashboard, and a human closes the loop by acknowledging from a phone — which flows back through the hub to the node.

This Repo — QSense Hub

The Snapdragon side of QSense Factory: a FastAPI hub + React dashboard, with all computer vision running on the Hexagon NPU via onnxruntime-qnn (Windows ARM64 — no OpenCV, no torch, no CUDA).

What it does

• PPE compliance detection on a live camera feed — per-person hard-hat / safety-vest status, annotated MJPEG stream, evidence snapshots on violation.
• MQTT alert brain — bridges qsense/machine/monitoring severity into warning/critical alerts with debounce, cooldown, hold-down auto-clear, and an ack/resolve lifecycle audited to SQLite.
• Downtime records — every machine alert becomes a downtime row (detection → acknowledgement), summarized as a candle chart and a per-machine table.
• Live dashboard — Overview / Live Monitor / Alerts / Settings / per-machine detail (live severity chart, anomaly log, downtime table), served by the hub and streamed over MQTT-WebSockets to PCs and phones alike.
• Runtime-tunable detection — score threshold, PPE requirements, severity thresholds and alert timing all hot-reload from the Settings page, no restart.

Minimum Workflow

USB camera ──ffmpeg──► PPE pipeline (Hexagon NPU) ──► /stream/camera (MJPEG)
                                   │
                                   ▼ qsense/ppe/snapshot (2 Hz)
QSense-Node ──qsense/machine/monitoring──►  Mosquitto (:1883 TCP / :9001 WS)
                                   │                        ▲
                                   ▼                        │ live (mqtt.js)
                        FastAPI hub (:8000) ──alerts──► Dashboard / phone
                                   └─ SQLite (history via REST /api/*)

Detailed App Flow

              ┌────────────────────────── Snapdragon X Elite (this repo) ─────────────────────────┐
              │                                                                                   │
 USB camera ──┼─► ffmpeg (dshow, 640×480@10) ─► FrameGrabber ─► Pipeline thread                   │
              │                                                   │  YOLOv8-PPE (NPU, ~34 ms)     │
              │                                                   │  face_det_lite (NPU, ~9 ms)   │
              │                                                   │  track ► classify ► annotate  │
              │                                                   ├─► FrameHub ─► MJPEG stream    │
              │                                                   ├─► ppe/snapshot ─► MQTT + DB   │
              │                                                   └─► ≥3 s violation ─► PPE alert │
              │                                                                                   │
 QSense-Node ─┼─► qsense/machine/monitoring ─► MqttBridge ─► AlertManager                         │
 (UNO Q)      │   qsense/machine/health      (severity ≥ warn/crit thresholds,                    │
              │   qsense/machine/anomaly      auto-clear after 10 s below warn,                   │
              │   qsense/machine/ack          offline after 30 s of silence)                      │
              │                                    │                                              │
              │                                    ├─► qsense/alert/event, machine/<id>/status    │
              │                                    ├─► downtime record (started → acked)          │
              │                                    └─► SQLite: alerts, events, telemetry, acks    │
              │                                                                                   │
 Dashboard ◄──┼── MQTT-WS :9001 (live)  +  REST /api/* (history/config)  +  MJPEG /stream/camera  │
 & phone   ───┼─► qsense/alert/action (ack / resolve)                                             │
              └───────────────────────────────────────────────────────────────────────────────────┘

AI Models — PPE Compliance on the NPU

Two ONNX models run per frame on the Hexagon NPU (QNN execution provider, HTP backend):

Model	Task	Input	Precision
YOLOv8-PPE (Hexmon/vyra-yolo-ppe-detection, YOLOv8m fine-tune, CC-BY-4.0)	PPE item detection, 14 classes	640×640 RGB	float
face_det_lite (Qualcomm AI Hub Models)	Person anchoring by face	640×480 gray	w8a8

Why YOLOv8-PPE?

We started with Qualcomm AI Hub's purpose-built PPE model, GearGuardNet (w8a8 / w8a16 exports still in ppe-detection/), and hit its limits fast:

• Only 2 classes (helmet, vest) — no explicit absence signal, so "no helmet detected" is indistinguishable from "helmet missed by the detector". We had to bolt on per-class threshold overrides (helmet 0.60 to kill bare-head false positives) and a second zoomed inference pass for distant workers.
• Quantized weights cost accuracy on our real camera at range.

The YOLOv8m PPE fine-tune fixed both structurally:

• Explicit NO-Hardhat / NO-Safety Vest classes — an explicit negative detection is a much stronger signal than merely failing to find a positive, and compliance.py lets an absence class override a same-region positive.
• 14 classes (Hardhat, Safety Vest, Gloves, Goggles, Mask, Fall-Detected, …) leave headroom for more safety rules without a new model.
• Float on the NPU — the HTP backend runs fp models fine, so we keep full accuracy and still get the ~6× NPU speedup (metrics below).

One caveat we found in its training data (confusion_matrix.png): the Person class is severely under-represented (251 validation instances vs 8254 for Hardhat), making it unreliable. So we ignore Person entirely and anchor people with face_det_lite instead — every detected face becomes a person, and PPE items are matched geometrically to head/torso regions derived from the face box.

Parameter changes we made (and what they did)

Parameter	Value (was)	What it did
SCORE_THRESHOLD	0.35 (0.45 GearGuardNet reference)	Recovers small/distant PPE items the higher gate dropped; hot-tunable from Settings (config.json reloads every 1 s, no restart)
NMS IOU_THRESHOLD	0.45 (0.7 reference)	Collapses duplicate boxes the looser reference value let through per object
Person class	ignored	Under-trained class produced phantom/missed people; face_det_lite anchors persons instead
Letterbox preprocessing	pad, not stretch	Preserves aspect ratio into the fixed 640×640 input — undistorted boxes at 640×480 capture
face NMS_IOU_THRESHOLD	0.3	Dedupes adjacent heatmap cells that tie at the same quantized score (w8a8 artifact)
face box enlarge/shift	+10% / 5%	Matches the qai_hub_models reference app — covers more face area so head-region matching is stable
Tracking persist_seconds	1.0 s (TRACK_IOU 0.3)	Detection boxes survive per-frame confidence flicker instead of blinking in and out
Compliance geometry	head: face ±0.3w, −1.0h…+0.2h · torso: ±0.8w, 5h below · overlap ≥ 0.15	Encodes "hard hat sits above the face, vest on the torso below it" — a hit anywhere else doesn't count
PPE alert debounce / cooldown	3 s / 30 s	One alert per sustained violation episode instead of one per frame; floor clear for 3 s auto-resolves
require_helmet / require_vest	independent toggles	Compliance rule matches the actual site policy (e.g. helmet-only zones) without redeploying

Engineering Metrics

Measured on this hub (Snapdragon X Elite, Windows 11 ARM64, Python 3.11 ARM64, onnxruntime-qnn, 2026-07-12). Latencies are full detect() calls — pre-process, NPU inference, and NumPy NMS post-process included:

Stage	NPU (Hexagon HTP)	CPU (ARM64)	Speedup
YOLOv8-PPE 640×640	33.7 ms median (36.2 mean / 52.6 max)	174.1 ms median (201.3 mean)	~5–6×
face_det_lite 640×480	9.1 ms median	6.7 ms median	~1× (tiny quantized model; NPU keeps it off the CPU cores)
Session init (one-time QNN graph compile)	7.9 s + 1.8 s	~0.1 s	paid once at startup

Per-frame vision budget ≈ 43 ms → the 10 FPS camera loop runs with >2× headroom (≈23 FPS theoretical). CPU-only would cap at ≈4.8 FPS while pegging cores; on the NPU the whole hub idles at a fraction of one core:

Process	RAM (working set)	CPU (steady state)
Hub (python run.py — capture, both models, annotate, FastAPI, MQTT, SQLite)	≈ 237 MB	≈ 0.21 cores avg (~2% of the X Elite)
Mosquitto broker	≈ 19 MB	negligible
ffmpeg capture (subprocess)	small	fixed decode cost of 640×480@10 MJPEG

Data-path rates: annotated JPEG (q80) published to the MJPEG stream every frame; PPE snapshots at 2 Hz over MQTT + SQLite; machine DB upserts throttled to 10 s; dashboard MQTT-WS reconnect every 2 s when the broker drops.

MQTT Topics

All traffic rides the local Mosquitto on the hub PC — devices over TCP :1883, browsers over WebSockets :9001 (listeners in server/mosquitto/qsense.conf).

Topic	Direction	Purpose
qsense/machine/monitoring	node → hub	severity stream, 1 Hz
qsense/machine/health	node → hub	lightweight heartbeat; keeps a quiet machine from going "offline"
qsense/machine/anomaly	node → dashboard	anomaly reports (same schema as monitoring); shown live in the machine page's Anomaly Log
qsense/machine/ack	node/hub ↔ hub	resolves the active alert for a machine_id, from any publisher
qsense/ppe/snapshot	hub → all (retained)	people / compliant / per-person PPE, 2 Hz
qsense/alert/event	hub → all	alert created / ack / resolved
qsense/machine/<id>/status	hub → all (retained)	healthy / warning / critical / offline
qsense/alert/action	dashboard → hub	ack / resolve from the UI
qsense/machine/<id>/cmd	hub → node	reserved for the closed loop (off by default)

Monitoring / anomaly payload:

{"alertId": "e2d69c69-…", "machineNo": "M-01", "partName": "Fan Motor",
 "partNo": "PN-001", "severity": 48.896, "timestamp": "2026-07-11T17:57:05.4"}

Alert Lifecycle & Severity

Severity score	Status	Dashboard	Downtime record
below warning threshold	Healthy	green card	—
≥ warning (default 20)	Warning	amber card + alert	opens on alert creation
≥ critical (default 40)	Critical	red card + alert escalates	continues the same record
no message for 30 s	Offline	grey card	—

Alerts auto-clear once severity holds below the warning threshold for ~10 s. A downtime record measures anomaly detection → acknowledgement; both thresholds are tunable live on the Settings page.

Hardware Requirements

Component	Purpose	Quantity
Snapdragon X Elite Copilot+ PC (Windows 11 ARM64)	Hub: NPU vision + broker + dashboard	1
USB / built-in camera	PPE compliance feed	1
Arduino UNO Q running QSense-Node	Machine vibration monitoring	1 per machine
Phone / tablet on the same LAN	Floor-side alert handling	any

Software Requirements

• Python 3.11 ARM64 venv at ppe-detection/.venv — onnxruntime-qnn, fastapi, uvicorn (no [standard] — httptools/uvloop lack ARM64 wheels), paho-mqtt, pillow, numpy
• Node.js LTS ARM64 (dashboard build)
• Eclipse Mosquitto 2.x as a Windows service, with both listeners from server/mosquitto/qsense.conf (TCP 1883 + WebSockets 9001) in its config
• ffmpeg (BtbN winarm64 build, on PATH) — camera capture; there is no OpenCV on ARM64
• Model files (gitignored, re-download): Hexmon/vyra-yolo-ppe-detection → ppe-detection/yolov8n-ppe/best.onnx, face_det_lite (w8a8) from Qualcomm AI Hub

Project Structure

VibeCheck/
├─ ppe-detection/          # CV package (also standalone-runnable)
│  ├─ yolo_ppe.py          #   YOLOv8-PPE pre/post-processing, class list, thresholds
│  ├─ face_detector.py     #   face_det_lite decode (dequant, peaks, NMS)
│  ├─ compliance.py        #   face + PPE boxes -> per-person helmet/vest verdict
│  ├─ ppe_detector.py      #   ONNX session factory (QNN EP), NMS, GearGuardNet path
│  ├─ live_detect.py       #   standalone Tkinter live viewer (dev tool)
│  ├─ yolov8n-ppe/         #   YOLOv8-PPE export (best.onnx, gitignored)
│  └─ face_det_lite-onnx-w8a8/, gear_guard_net-onnx-*/   # AI Hub exports
├─ server/                 # FastAPI hub
│  ├─ run.py               #   entry point (:8000)
│  ├─ pipeline.py          #   camera -> NPU models -> track -> classify -> publish
│  ├─ mqtt_client.py       #   MqttBridge: subscriptions + publishes
│  ├─ alerts.py            #   AlertManager: thresholds, lifecycle, downtime, status
│  ├─ tracking.py          #   IoU tracking + overlay drawing
│  ├─ camera.py            #   ffmpeg dshow -> MJPEG frame grabber
│  ├─ config.py/.json      #   hot-reloaded runtime tunables (Settings page)
│  ├─ db.py                #   SQLite: telemetry, alerts, events, downtime, acks
│  └─ mosquitto/qsense.conf#   broker listeners (TCP 1883 + WS 9001)
├─ dashboard/              # React + Vite admin UI -> dist/ served by the hub
│  └─ src/pages/           #   Overview, LiveMonitor, Alerts, Settings, MachineDetail
├─ docs/                   # README assets
└─ start_all.ps1           # build dashboard + launch hub

CLI Quick Reference

Action	Command
Start everything	.\start_all.ps1
Hub only	ppe-detection\.venv\Scripts\python.exe server\run.py
Dashboard dev server (hot reload, proxies to :8000)	cd dashboard; npm run dev
Rebuild the dashboard the hub serves	cd dashboard; npm run build
Inject a machine reading (trips critical)	mosquitto_pub -h 127.0.0.1 -t qsense/machine/monitoring -m '{"alertId":"t1","machineNo":"M-01","partName":"Fan Motor","partNo":"PN-001","severity":45,"timestamp":"2026-07-12T00:00:00"}'
Inject an anomaly-log entry	mosquitto_pub -h 127.0.0.1 -t qsense/machine/anomaly -m '{"machineNo":"M-01","severity":5.2,"partName":"Fan Motor","partNo":"PN-001"}'

Override brokers with QSENSE_MQTT_HOST (hub) and VITE_MQTT_URL (dashboard, e.g. ws://<pc-ip>:9001 — rebuild after changing).

How to Run

1. Make sure the Mosquitto service is running with both listeners (TCP 1883 + WebSockets 9001; copy them from server/mosquitto/qsense.conf into the service's mosquitto.conf once, then restart the service).
2. Launch the hub — this also rebuilds the dashboard it serves:

   .\start_all.ps1

3. Open http://localhost:8000/. The MQTT LIVE badge (top right) confirms the browser's WebSocket leg; LIVE · NPU on Live Monitor confirms the vision pipeline is on the Hexagon NPU.
4. For phones, allow the port once in an elevated PowerShell, then browse to http://<pc-ip>:8000/:

   netsh advfirewall firewall add rule name="QSense" dir=in action=allow protocol=TCP localport=8000

5. Point QSense-Node devices at the hub PC's IP (e.g. 192.168.8.153:1883). No device handy? Use the mosquitto_pub one-liners above.
6. Demo: stand in front of the camera without a hard hat → Live Monitor flags missing helmet, ~3 s later a PPE alert appears with an evidence snapshot. Cross the severity threshold on a machine → its card escalates live; open the machine page for the live chart, anomaly log, and downtime table; ack from a phone and watch the downtime record close.

How it Works

Vision — server/pipeline.py

A daemon thread pulls the newest JPEG from ffmpeg (FrameGrabber keeps a 1-frame queue — always fresh, never backlogged), runs YOLOv8-PPE + face_det_lite on the NPU, matches PPE boxes to head/torso regions per face (compliance.py), tracks boxes across frames to kill flicker (tracking.py), draws the overlay, and publishes: JPEG → FrameHub (MJPEG endpoint), snapshot → MQTT + SQLite at 2 Hz. A violation sustained ≥ 3 s writes an evidence JPEG and raises a PPE alert; a floor clear for the same window auto-resolves it. Config hot-reloads every second, and the camera restarts itself after 5 s without frames.

Alert brain — server/mqtt_client.py + server/alerts.py

One paho-mqtt client subscribes to monitoring / health / ack / action topics. AlertManager turns severity readings into warning/critical alerts (create, escalate, 10 s hold-down auto-clear), derives machine status (active alert > fresh reading > 30 s silence = offline), opens a downtime record per alert and closes it on acknowledgement, and audits everything (alerts, acks, events, telemetry) to SQLite via db.py.

Dashboard — dashboard/

React + Vite. lib/live.tsx is one MQTT-over-WebSockets client feeding a context: machine cards, live severity charts, the PPE widget, alert banners and the per-machine anomaly log all update push-style, no polling. REST (lib/api.ts) seeds history (telemetry, compliance, downtime) so charts survive a page reload; the camera is a plain MJPEG <img>. Ack/resolve buttons publish qsense/alert/action straight onto the broker.

The Closed Loop

   UNO Q senses vibration ──► severity ──► Hub thresholds ──► alert + downtime
        ▲                                                            │
        │                                                            ▼
   (cmd topic reserved:                                    Dashboard on the wall
    auto-stop on critical)                                 + phone on the floor
        ▲                                                            │
        └───────────── qsense/machine/ack ◄── supervisor acknowledges┘

License

Hackathon project by team VibeCheck-Q (Snapdragon Multiverse Hackathon). Bundled model weights keep their own licenses: Hexmon/vyra-yolo-ppe-detection (CC-BY-4.0), face_det_lite / GearGuardNet from Qualcomm AI Hub Models (BSD-3-Clause).