robot_tracking_server

Overhead perception server for the HamBot system. Runs on a Jetson Nano connected to a ceiling-mounted Intel RealSense D435 camera. Detects robots in the field using ArUco markers and streams world state JSON to all connected robot clients over TCP.

---

How the System Works

HamBot operates as a two-component system:

┌──────────────────────────────────┐         ┌──────────────────────────────────────┐
│          Jetson Nano             │   TCP   │              HamBot                  │
│                                  │ ──────► │                                      │
│  RealSense D435 (ceiling mount)  │  JSON   │  world_state_receiver.py             │
│  Detects all robots in field     │  ~30Hz  │    background thread caches latest   │
│  Streams world state to clients  │         │                                      │
│                                  │         │  receiver.get()  ← behavior code    │
└──────────────────────────────────┘         └──────────────────────────────────────┘

This repo — runs on the Jetson Nano. Detects robots via ArUco markers with an HSV color fallback, and broadcasts a world state JSON packet to every connected robot simultaneously.

HamBot Client — runs on each robot's Raspberry Pi. A background thread receives the continuous stream from the Jetson Nano and caches the latest packet. Behavior code calls receiver.get() on demand to pull the current world state. See: robot_tracking_client

---

Requirements

• Jetson Nano (Linux)
• Python 3.11+
• Intel RealSense D435 camera
• Intel RealSense SDK 2.x — install before pip packages: https://github.com/IntelRealSense/librealsense/releases

---

Installation

git clone https://github.com/biorobaw/robot_tracking_server.git
cd robot_tracking_server

python -m venv venv
source venv/bin/activate

pip install -r requirements_server.txt

Note: Use opencv-contrib-python, not opencv-python. The contrib package includes the ArUco module required for marker detection.

---

Quick Start

# Two robots, no goal
python world_state_server_nano.py --marker-ids 1 7

# Three robots with goal position
python world_state_server_nano.py --marker-ids 1 7 42 --goal-x 110 --goal-y 0

# Custom camera height (default is 220 cm)
python world_state_server_nano.py --marker-ids 1 7 --camera-height 185

# Headless (no display window)
python world_state_server_nano.py --marker-ids 1 7 --no-display

Robots can connect and disconnect at any time — the server keeps running and detecting without interruption.

---

Arguments

Argument	Default	Description
--marker-ids	required	ArUco marker IDs to track (space-separated)
--host	0.0.0.0	IP to bind (all interfaces)
--port	9999	TCP port (must match client)
--goal-x	none	Goal X position in cm (optional)
--goal-y	none	Goal Y position in cm (optional)
--camera-height	220.0	Camera height above floor in cm
--no-display	off	Run headless, no OpenCV window

--goal-x and --goal-y must be provided together or not at all.

---

Repository Structure

robot_tracking_server/
├── world_state_server_nano.py   # TCP server — run this
├── world_state_nano.py          # Multi-robot world state estimator
├── aruco_detector.py            # ArUco marker detection
├── hsv_detector.py              # HSV color detection (robot fallback)
├── camera.py                    # RealSense camera wrapper
├── hsv_profiles.json            # Saved HSV tuning profiles
├── requirements_server.txt
└── README_server.md

---

Robot Detection

Each robot is identified by a unique ArUco marker (DICT_4X4_50). The server tracks all configured marker IDs every frame.

Detection priority per robot:

1. ArUco detected → full position + fresh heading
2. ArUco lost → nearest HSV green blob fills position, heading held from last ArUco fix
3. No HSV blob found → robot marked lost, last known position held

HSV is never used to assign robot identity — only to fill position for a robot already identified by ArUco. This prevents blob-swap errors when two robots are close together and both lose ArUco in the same frame.

Note: A heading staleness warning is shown in the display HUD when a robot has been on HSV fallback for more than 10 frames without a fresh ArUco heading.

---

HSV Tuning

HSV profiles are saved in hsv_profiles.json. The hambot_green profile is included and tuned for the HamBot robot body color under typical lab lighting.

To retune for your lighting conditions, run the HSV detector standalone:

python hsv_detector.py

Press 2 to tune the robot green profile, s to save.

---

Finding the Jetson Nano's IP Address

Robots need to know the Jetson Nano's IP to connect.

ip addr
# Look for the IP on your WiFi or Ethernet adapter

---

License

MIT