hero_evaluation_benchmarking

This repository contains the evaluation package for benchmarking of connectivity aware and battery aware algorithms.

How to run

1. Bind the package to the competition container:

This package has been tested within the competition Docker container. To use it, bind the package directory to ~/ros2_ws/src inside the container. To do so, add the following line to the first_run.sh file as an arugment to the docker run command:

docker run -it\
.
--volume "<evaluation_dir_host_machine>:/root/CrazySim/ros2_ws/src/icuas26_evaluation:rw"\
.

2. Build:

./src/icuas26_evaluation/launch/setup_icuas26_evaluation.sh

This script copies the worlds from icuas26_evaluation/worlds folder to icuas26_competition/worlds folder, and builds both the packages.

This script automatically detects if the workspace was built with --merge-install or not and respects that.

3. Run

You can start the evaluaton scrips by running:

./ros2_ws/src/icuas26_evaluation/startup/start.sh

This launches the main launch file for evaluation.

Available parameters can be set in config/eval_params.yaml. Also, remember to set team name and other stuff in icuas26_competition/startup/_setup.sh

Crash check

Manually note if there was any false positive crash logs, and remove them from txt later. Conversely, if there are crashes that didn't register, you can add that to txt later in this format: 23.84: UAVs ['cf_1'] have collided with an obstacle.. You can put estimates timstamp.

You can modify the corresponding json by running: python3 scripts/sync_json_from_txt.py results/log_aeras_icuas26_1_5_70.0.txt

• If a uav crashes, its position starts glicthing between crash position and base. So it says entered base, and if all crash and/or are in base and mission done is received logs will terminate

0.23: UAVs ['cf_1'] left the base.
0.23: UAVs ['cf_1'] landed outside the charging area.
1.03: UAVs ['cf_1'] entered the base.
1.23: UAVs ['cf_1'] left the base.
1.23: UAVs ['cf_1'] landed outside the charging area.
1.43: UAVs ['cf_1'] entered the base.
1.83: UAVs ['cf_1'] left the base.
1.84: UAVs ['cf_1'] landed outside the charging area.
2.83: UAVs ['cf_1'] entered the base.

---

---

Evaluation Metrics & Logged Events

The following events are automatically logged during mission execution:

Code	Category	Event	Description
C1	Connectivity	Bifurcation	Network splits into multiple components (no single agent isolated)
C2		Agent Disconnected	Specific agent(s) lost connectivity
C3		Network Reconnected	Connectivity restored (Fiedler value > 0)
C4		Disconnection Timeout	Network remained disconnected for disconnection_timeout seconds; mission force-terminated
B1	Base Station	UAV Entered Base	UAV lands in the charging area
B2		UAV Left Base	UAV takes off from charging area
B3		Landed Outside Valid Areas	UAV landed (z ≤ 0.25 m) but outside both the charging area and all landing pads
B4		Landed On Landing Pad	UAV landed on a landing platform pad (derived from landing aruco markers)
E1	Environment Bounds	Entered Bounds	Agent enters the defined environment volume
E2		Left Bounds	Agent exits the environment bounds
A1	Aruco Detection	Target Marker Found	Target marker (ID target_id_min–target_id_max) detected, distance from GT logged
A2		Landing Platform Marker Found	Landing platform marker (ID landing_id_min–landing_id_max) detected, distance from GT logged
A3		Decoy Marker Found	Decoy marker (ID > landing_id_max) detected, distance from GT logged
A4		Marker Not In World	Detected Aruco ID doesn't exist in ground truth
X1	Collision	Obstacle Collision	UAV came within collision_radius of an occupied octomap voxel (logged once per UAV)
T1	Battery	Critical Battery	UAV battery drops below min_battery threshold
M1	Mission Status	Mission Timeout	Mission exceeds mission_timeout parameter
M2		Mission Complete	mission_done received and all UAVs in base
M3		Mission Pending	mission_done received but UAVs still flying
W1	Warnings	Hardware Warning	Pose/image timeout, velocity exceeded, or proximity warning

Log Output: results/log_{team}_{env}_{robots}_{range}.txt (and .json if enabled)

RViz Visualization Quick Reference

Element	Visual		Element	Visual
UAVs	White sphere (brightness = battery %)		Charging base	Red box
Landing pads	Yellow flat box		Battery bars	White cylinders + text
Target markers (GT)	Red cube		Detected markers	Green cube
Landing markers (GT)	Yellow cube		Edges (connected)	Green line
Decoy markers (GT)	Orange cube		Edges (out of range)	Orange line
LOS range	Yellow transparent sphere		Edges (obstructed)	Red line

---

Manual Testing Checklist

Run each scenario and verify the corresponding log entry appears correctly.

Vsiualization:

• Battery levels
• Edges

Base Station Events:

• UAV Entered Base - Land a UAV in the charging area, verify log shows correct UAV ID
• UAV Left Base - Take off from charging area, verify log shows correct UAV ID
• Multiple UAVs entering/leaving base simultaneously
• Landed On Landing Pad - Land UAV on a landing platform, verify B4 logged
• Landed Outside Valid Areas - Land UAV outside both base and pads, verify B3 logged

Environment Bounds Events:

• UAV Entered Bounds - Fly UAV into defined environment volume
• UAV Left Bounds - Fly UAV outside environment bounds, verify warning logged

Connectivity Events:

• Network Disconnected - Move UAVs apart until Fiedler value = 0, verify log
• Network Reconnected - Bring UAVs back into range, verify reconnection logged
• Bifurcation Detected - Split network into multiple components
• Agent Disconnected - Verify specific UAV IDs logged when losing connectivity

Battery Events:

• Critical Battery - Let battery drop below MIN_BATTERY threshold, verify log with UAV ID
• return_to_base topic publishes True when any battery is critical (hardware mode only)

Aruco Detection Events:

• Target Marker Found - Detect Aruco in target range, verify A1 logged with distance
• Landing Platform Marker Found - Detect Aruco in landing range, verify A2 logged with distance
• Decoy Marker Found - Detect Aruco in decoy range, verify A3 logged with distance
• Marker Not In World - Report non-existent Aruco ID, verify A4 logged

Mission Status Events:

• Mission Timeout - Let mission exceed mission_timeout, verify timeout logged
• Mission Complete - Send mission_done with all UAVs in base, verify completion logged
• Mission Pending - Send mission_done with UAVs still flying, verify pending status logged

Warning Events (Hardware Mode Only):

• Pose Timeout - Stop pose updates for > timeout_threshold seconds
• Image Timeout - Stop image updates (simulates AI deck disconnection)
• Velocity Exceeded - Move UAV faster than MAX_VEL
• Proximity Warning - Fly two UAVs closer than MIN_HOR_DIST on XY plane

Mode-Specific Tests:

• mode: "hardware" - Verify battery reads from /cf_{i}/status (voltage)
• mode: "sim" - Verify battery reads from /cf_{i}/battery (percentage)
• mode: "hardware" - Verify hardware_safety_node launches
• mode: "sim" - Verify hardware_safety_node does NOT launch

---

Event Log Examples

JSON entries for each event code. Examples without a marker are taken directly from test run logs (log_LARICS_NONE_3_70.0 / log_LARICS_NONE_5_2.5). Examples marked † are constructed from code output for codes not triggered in those runs.

---

Connectivity

C1 — Network bifurcation †

{
  "timestamp": 426.03,
  "type": "connectivity",
  "code": "C1",
  "message": "Network has disconnected. Info: Network bifurcation."
}

C2 — Agent disconnected

{
  "timestamp": 1.38,
  "type": "connectivity",
  "code": "C2",
  "message": "Network has disconnected. Info: AGV disconnected.",
  "agents": ["AGV"]
}

C3 — Network reconnected

{
  "timestamp": 7.01,
  "type": "connectivity",
  "code": "C3",
  "message": "Network has reconnected."
}

C4 — Disconnection timeout, mission force-terminated

{
  "timestamp": 86.41,
  "type": "connectivity",
  "code": "C4",
  "message": "Network disconnected for 60s. Force-terminating mission."
}

---

Base Station

B1 — UAV entered base †

{
  "timestamp": 38.03,
  "type": "base",
  "code": "B1",
  "message": "UAVs ['cf_1'] entered the base.",
  "agents": ["cf_1"]
}

B2 — UAV left base

{
  "timestamp": 0.98,
  "type": "base",
  "code": "B2",
  "message": "UAVs ['cf_1'] left the base.",
  "agents": ["cf_1"]
}

B3 — UAV landed outside all valid areas †

{
  "timestamp": 299.43,
  "type": "base",
  "code": "B3",
  "message": "UAVs ['cf_1'] landed outside the charging area and landing pads.",
  "agents": ["cf_1"]
}

B4 — UAV landed on a landing platform †

{
  "timestamp": 310.20,
  "type": "base",
  "code": "B4",
  "message": "UAV cf_1 landed on landing platform (marker ID 15).",
  "agents": ["cf_1"],
  "platform_marker_id": 15
}

---

Environment Bounds

E1 — Agent entered environment bounds †

{
  "timestamp": 12.50,
  "type": "bounds",
  "code": "E1",
  "message": "UAVs ['cf_1'] has entered the environment bounds.",
  "agents": ["cf_1"]
}

E2 — Agent left environment bounds †

{
  "timestamp": 510.63,
  "type": "bounds",
  "code": "E2",
  "message": "UAVs ['cf_1'] has left the environment bounds.",
  "agents": ["cf_1"]
}

---

Aruco Detection

Three marker types are distinguished by configurable ID ranges (set in eval_params.yaml):

Range	Type	Default IDs
target_id_min–target_id_max	Target	0–10
landing_id_min–landing_id_max	Landing platform	11–20
> landing_id_max	Decoy	21+

A1 — Target marker detected †

{
  "timestamp": 150.18,
  "type": "aruco",
  "code": "A1",
  "message": "Target marker ID 7 found at [1.0, 1.0, 1.0]. Distance from GT: 5.039.",
  "aruco_id": 7,
  "marker_type": "target",
  "distance": 5.039
}

A2 — Landing platform marker detected †

{
  "timestamp": 160.50,
  "type": "aruco",
  "code": "A2",
  "message": "Landing platform marker ID 15 found at [2.0, 3.0, 0.5]. Distance from GT: 0.120.",
  "aruco_id": 15,
  "marker_type": "landing",
  "distance": 0.12
}

A3 — Decoy marker detected †

{
  "timestamp": 170.33,
  "type": "aruco",
  "code": "A3",
  "message": "Decoy marker ID 25 found at [4.0, 2.0, 1.0]. Distance from GT: 0.045.",
  "aruco_id": 25,
  "marker_type": "decoy",
  "distance": 0.045
}

A4 — Marker not in ground truth †

{
  "timestamp": 99.75,
  "type": "aruco",
  "code": "A4",
  "message": "Aruco ID 99 found at [1.0, 1.0, 1.0]. There is no Aruco ID 99 in this world.",
  "aruco_id": 99
}

---

Collision

X1 — UAV collided with an obstacle †

{
  "timestamp": 42.15,
  "type": "collision",
  "code": "X1",
  "message": "UAVs ['cf_2'] have collided with an obstacle.",
  "agents": ["cf_2"]
}

---

Battery

T1 — Battery below critical threshold

{
  "timestamp": 0.98,
  "type": "battery",
  "code": "T1",
  "message": "UAVs [1] have breached the minimum battery threshold.",
  "agents": [1]
}

---

Mission Status

M1 — Mission timeout †

{
  "timestamp": 60.0,
  "type": "mission",
  "code": "M1",
  "message": "______________________Mission timeout (1m) - Log terminated______________________"
}

M2 — Mission complete, all UAVs in base †

{
  "timestamp": 300.0,
  "type": "mission",
  "code": "M2",
  "message": "Mission done received, all UAVs in base, ending run."
}

M3 — Mission done received, UAVs still flying †

{
  "timestamp": 250.0,
  "type": "mission",
  "code": "M3",
  "message": "Mission done received, all UAVs not in base"
}

---

Warnings

W1 — Hardware warning (pose/image timeout, velocity, proximity) †

{
  "timestamp": 45.0,
  "type": "warning",
  "code": "W1",
  "message": "cf_1: Pose timeout (2.3s)"
}

---

Dev notes

edge_evaluator_node.cpp creates a service of type ComputeEdges (custom):

geometry_msgs/Point[] positions
---
geometry_msgs/Point[] positions
float32[] batteries
icuas26_evaluation/Edge[] edges
bool[] collisions

It subscribes to all relevant cf topics, creates the adjacency matrix and edges in the required format (see below). Additionally, it can be called with a positions argument, which can be augmented to the computation of edges. This is used by the eval_node.py that sends the base as an additional node of the graph, when it calls this service and gets all positions, edges (including base) and batteries. After getting these, the eval node does two things: (1) Logging, mainly handled by the Logger class (/icuas26_evaluation/modules/logger.py); (2) visualization managed by the VisualizationManager (/icuas26_evaluation/modules/logger.py) class

All colors, shapes and sizes of markers can be modified in the VisualizationManager class. Also, since the battery voltages are visualized in the 3D scene itself, they might overlap with the flying arena. This can be changed by changing the origin variables in the vis_battery_levels() of the class.

The _setup.sh file must be sourced and should be in the following format.

The min_battery parameter is configured in config/eval_params.yaml and is used by hardware_safety.py (in hardware mode) to determine when to publish the return_to_base msg flag and by the eval node to normalize battery colors for better visualization. Remember that this is 'flying voltage'.

All paths related to configs and binvox are set in launch.py file and can be changed accordingly.

Aruco visualization: Orientations are not specified in the TargetFound topic, so only the position of Arcos is visualizable.

Some considerations:

• If the floor of octomap is too thick, the UAV positions on landing might be occupied. To avoid this, keep the floor thin (what Marko set it as in last run).
• There is a variable called self.height_sensitivity in Base class (/icuas26_evaluation/modules/logger.py). This denotes the height after which the UAV is logged as it has left the base. In case the landing position of UAVs is higher than the floor, this variable can be increased.
• It is asummed that Arucos csv file (config/coordinates_aruco.csv) is in order of Aruco IDs (first line is ID 0, second is 1...)

Sim-specific tricks

• The binvox .bt file can sit slightly above ground level. This causes landed UAVs to appear as if they are inside obstacles, making them seem disconnected in the edge evaluator. To handle this, UAV z positions are clamped to a minimum of 0.25m in edge_evaluator_node.cpp.

---

Collision Detection

Two independent methods are available, selected via collision_method in eval_params.yaml. Both emit X1 events through the same path (C++ → ComputeEdges response → Python logger). A crash is a one-time event per UAV — once logged it is never repeated regardless of how many times the check fires.

Method 1: Proximity (collision_method: "proximity")

Checks whether any occupied octomap voxel is within collision_radius of the UAV on every service call. is_near_obstacle() iterates a 3D grid centred on the UAV position with step size equal to the octomap resolution (no voxels skipped), checks the euclidean distance to prune grid corners, and queries octree_->search() + isNodeOccupied().

for x, y, z in grid(pos ± radius, step=octomap_resolution):
    if z <= min_obstacle_z:             skip   ← floor filter
    if distance(pos, (x,y,z)) > radius: skip   ← sphere prune
    if octree.search(x,y,z) occupied:  return true
return false

Good for: detecting collisions with walls/pillars/obstacles at any flight altitude. Requires a well-built octomap with correctly placed obstacle voxels.

Method 2: Glitch (collision_method: "glitch")

After a heavy crash in simulation the UAV's pose topic alternates rapidly between the actual crash position and (0, 0, 0) world origin. This check counts how many times the raw pose (before any z-clamping) lands within glitch_origin_radius of origin. Once the count reaches glitch_threshold the UAV is permanently flagged as crashed.

on each pose message (raw, before z-clamp):
    if distance_to_origin(raw_pos) < glitch_origin_radius:
        origin_visit_count[uav]++
        if origin_visit_count[uav] >= glitch_threshold:
            glitch_crashed[uav] = true

Good for: catching heavy crashes that are hard to detect geometrically (e.g. UAV clips a wall at high speed and the sim physics engine goes unstable). Does not require octomap accuracy.

Caveat: if the charging base is positioned very close to world origin (within glitch_origin_radius), landing there will accumulate counts. Keep glitch_origin_radius small (default 0.05 m) and ensure the base is not at origin.

Method 3: Both (collision_method: "both")

Either condition triggers the crash flag. Proximity fires first; glitch is checked only if proximity returned false (short-circuit). Useful during development to compare the two signals.

Parameters (config/eval_params.yaml)

Parameter	Default	Method	Effect
collision_method	"proximity"	—	Which check(s) to run: "proximity", "glitch", or "both"
collision_radius	0.15	proximity	Sphere radius (m) around UAV to search for occupied voxels. Set to 0 to skip all octomap queries.
min_obstacle_z	0.3	proximity	Voxels at or below this z (m) are excluded — prevents floor voxels triggering false positives during takeoff.
glitch_threshold	5	glitch	Number of origin-snap events before a crash is registered. Higher = more evidence needed, slower detection.
glitch_origin_radius	0.05	glitch	Distance (m) from world origin within which a raw pose counts as a glitch snap. Keep small.

Known false-positive sources and fixes

Proximity — false positive during takeoff / near the floor When a UAV has just taken off (z ≈ 0.3–0.4 m), the floor voxels below can fall within collision_radius. The min_obstacle_z filter excludes those voxels. Default 0.3 m clears a typical thin floor mesh. If false positives persist, increase it — but also keep the world floor mesh thin (≤ one voxel layer).

Proximity — false positive for landed UAVs Any UAV with z ≤ 0.25 (the z-clamp threshold) skips the proximity check entirely and always reports false.

Glitch — base near world origin If the charging area is within glitch_origin_radius of (0,0,0), landing there will accumulate counts and eventually trigger X1. Either move the base in the world, or reduce glitch_origin_radius below the base–origin distance.

Disabling collision detection Set collision_method: "proximity" and collision_radius: 0 — the proximity check short-circuits immediately. Or set collision_method: "glitch" with glitch_threshold to a very large number.

Logging

The logs are saved in folder /results with file name as "log_Team_environment_numuavs_commrange.txt". Make sure to write the team name in _setup.sh file so it can be automatically grabbed by the evaluation node.

Visualization

The RViz configuration file is located in the config folder of this package. The following visual elements are included:

1. Base Visualization:

   • Represented as a red box.
   • The dimensions depend on the bounds of the charging area.

2. Agent Visualization:

   • Each Crazyflie is represented as a colored sphere.
   • The intensity of the sphere's color indicates its battery percentage.

3. Edge Visualization: Each communication link is assigned a status code:

   • 0: Valid (green)
   • 1: Line-of-sight but outside communication range (orange)
   • 2: Obstructed (red)

4. Line-of-Sight Range: A sphere representing the line-of-sight range of each agent is shown. Note: The surface area of these spheres may appear rough when the spheres are large, due to rendering limitations of RViz markers.

5. Battery Bar Plot: A plot is shown using cylinder markers and text in rviz. Previously, a bar plot image was used but that proved to be computationally expensive/

All colors used in the visualization can be modified in the visualization_manager.py file.

All topics are published under the namespace '/visualization/'