Conversation
Brings the scoring harness from scene-benchmark-harness onto current dev: the WBT ground truth, the one-to-one Hungarian matcher, the acceptance fixtures and the five-world runner. Only testing/ is taken; the Scene implementation stays as it is on dev, since the harness branch predates it by 212 commits and merging the two would have mixed a stale perception stack into the scores. The parse of the robot node had gone wrong in the meantime. `_vector_field` matches `^\s*translation ...` anywhere inside the node block, and e535894 added a cameraSlot as TiagoLite's first child. The first translation in the block became the camera's (-0.028, -0.035, -0.009) mount offset instead of the robot's (0.962, -4.801, 0.235) world pose. Every ground-truth centre is expressed relative to the robot's initial pose, so the whole fixture was displaced by about 4.9 m and any accuracy computed against it was meaningless — a silent failure, since the numbers still looked like plausible coordinates. `_own_fields` keeps a lookup on the node's own depth, and the office target resolves to 1.5464 again, matching the 1.54636 the fixture recorded before the regression. Two harness tests still fail against today's world file: the RGB optical frame was renamed to head_front_camera_rgb_optical_frame, and the window-rerank fixture lost its `config` key. Both are stale expectations in the harness rather than defects in the ground truth, and are left for a separate pass.
run_webots_scene_benchmark.sh defaults to robonix_manifest.scene-eval.yaml and exits immediately without it. The earlier restore took only testing/, so the harness could not start. This deployment drops pilot and executor and keeps the perception path, so scores measure Scene rather than whatever the planner happened to do that run.
system_listen returned None for a manifest that never set listen, so boot skipped the port pre-check and the component died on bind deep inside its own log, after boot had already reported the earlier stages OK. Fall back to each builtin's compiled-in default address so the check covers those manifests too. Also move the Webots Scene evaluation deployment's soma to 50191: the benchmark host runs a separate long-lived deployment that holds the default 50091, and scoring Scene must not require tearing that down.
The benchmark points ROBONIX_HOME at a /tmp directory but never wrote robonix_source_path into it. Every package spawned under that home reads the key to resolve capability and IDL paths and refuses to start without it, so a reboot — or a first run on a fresh machine — made all three primitives exit "before registering with atlas", several layers below the real cause. Write both keys before the first world starts, preserving any other keys already there.
… reference The URDF carries a `<gazebo>` block declaring a ground-truth odometry plugin, but Webots ignores `<gazebo>` entirely, so `/webots/ground_truth/odom` never existed. Every consumer saw an empty stream: the Scene benchmark's sweep read zero ground-truth samples, which made its visibility projection report one visible object out of fifty-one and its odometry agreement divide by a zero-length path. Add a webots_ros2_driver plugin that reads the robot's pose from the supervisor and publishes it, and register it in both Tiago URDFs. The plugin publishes nothing and warns when the robot is not a supervisor, rather than raising and costing the deployment its sensors to gain a diagnostic. This is a reference signal for scoring, not a localization source: navigation and mapping must not consume it, or it would hide the drift it exists to measure.
…its node The driver does not initialise rclpy on behalf of Python plugins, so create_node raised NotInitializedException on every boot. The plugin then had no `_node` attribute at all, and `step` raised AttributeError once per simulation step for the rest of the run — the intended graceful degradation never applied, because it checked an attribute that assignment had never reached. Initialise rclpy when no context is live, give the three attributes class-level defaults so a failed init degrades to a no-op step, and check the publisher before spinning rather than after.
…easured Every truth object came back with zero depth-consistent pixels, including one whose bbox projected across the entire frame, and the diagnostics could not distinguish the three causes that produce that number: an occluded object, an empty depth frame, or a projection placing the object at the wrong range. Record the expected depth interval alongside the observed frame's finite fraction and min/median/max, sampled from the largest projection seen for each object.
The diagnostics mixed the run's largest projected area with the last frame's rejection reason and depth interval, describing a frame that never occurred: a window that projected across 18981 px was reported as "projected_area_too_small", and the expected depth interval was always None because the last frame had rejected the object before computing one. Record reason, clipped fraction, consistent-pixel counts, expected interval and observed depth together from the frame where each object projected largest — the frame where it had its best chance of being seen.
…ered to The RangeFinder declared its own frame name, so the driver published head_front_camera_depth_optical_frame with identity rotation while the URDF's optical joint for it never reached the TF tree at all. Measured live: the RGB optical frame carries rpy=(-1.5708, 0, -1.5708) and the depth one carries rpy=(0, 0, 0), at byte-identical translations. A consumer that reads the frame from the depth image header and projects with the optical convention then applies a body-oriented transform: forward distance lands on Y and the height difference becomes the reported depth. That is why the Scene benchmark placed 48 of 51 truth objects "out of range" and the only three it could project were the three tallest — the only ones above the camera, hence the only ones with a positive "depth". The topic is depth_registered, so its frame is the colour camera's optical frame by definition. Scene was already unaffected because it selects the RGB optical frame explicitly; its potted plant lands 8 cm from the WBT truth.
Seven environment variables the detector reads were never passed into the container, so setting them on a docker deployment did nothing and the operator had no way to tell: SCENE_CG_MIN_POINTS, SCENE_CG_OBJ_MAX_POINTS, SCENE_CG_MERGE_THRESHOLD, SCENE_CG_VOXEL_SIZE, SCENE_CLIP_MODEL, SCENE_CLIP_PRETRAINED and SCENE_DETECT_PERIOD_S. Two of those — the point-count floors — are among the few knobs that move detection recall at all. Forwarding them unconditionally passes `-e NAME=` for the ones nobody set, which the numeric override table already ignores but the model selection did not: it would have loaded a model named "" and failed deep inside the library with an error naming no variable. Treat blank as absent there too.
Every other perception knob has an environment override; the one threshold that decides whether a detection exists at all did not, so changing it meant rebuilding the image. Upstream concept-graphs runs 0.2 against real imagery, low-texture synthetic scenes want room to go lower still, and a cluttered deployment may want it higher. Measured motivation: adding "window" to the open vocabulary produced zero window detections in a world containing eight of them, and the detector log shows the class was never reported at all — the words are necessary but the confidence floor is what binds.
…ction Localized-vs-truth agreement reported median, p95, max and final, which are nearly equal both when the estimator carries a constant offset from the first comparison and when it applies one correction early and then holds it. Those are different faults and the summary could not tell them apart. Record the minimum error and the point in the sequence where the error first reaches half its final magnitude: ~0.0 for an offset that was always there, the landing point for a correction applied later.
Component logs were copied only after a world finished scoring, so every run that failed discarded the scene, mapping and nav logs when the teardown trap stopped the deployment — the runs whose logs are the only record of why they failed were the ones that kept none. Two confidence-arm runs died at the localization gate tonight and left nothing to compare against the arm that passed. Capture from the trap as well, guarding the function against being called before the first world has a directory.
concept-graphs gates per-detection denoising behind `dbscan_remove_noise and run_dbscan`, and every call site here passed the second half as a constant false. The `dbscan_remove_noise` default of true therefore only ever reached the periodic pass over the accumulated map; a detection's own cloud was never denoised. That distinction decides more than storage. Mask edges bleed depth onto whatever lies behind the object, and those points inflate the detection's bbox and drag its centroid before the merge decision is taken — so association runs on geometry the periodic pass would later have cleaned, and a merge made on that geometry cannot be undone afterwards. Expose it as `per_detection_dbscan`, defaulting to false so behaviour is unchanged, with SCENE_CG_PER_DETECTION_DBSCAN to measure the trade against its per-detection clustering cost. The env parser rejects values that are not clearly boolean rather than reading any non-empty string as true, matching how the numeric overrides leave the default in place on a value they cannot parse.
…a scorer layout Inert unless SCENE_EXPORT_CG_PICKLE names a directory. When it does, the detector writes full_pcd_<exp>.pkl.gz atomically after each periodic cleanup and once more on stop, in the exact shape conceptgraph/scripts/eval_replica_semseg.py loads, so Scene's map can be scored by the paper's own evaluator without any conversion step. The export records which CLIP produced the object features so the scorer can pick a matching text encoder instead of assuming ViT-H-14.
SCENE_CAMERA_FRAME and SCENE_BASE_FRAME are read by the service but were never passed through start.sh, so setting them on the host had no effect. SCENE_EXPORT_CG_PICKLE and SCENE_EXPORT_CG_EXP are forwarded the same way, and the export directory is bind-mounted at its own path so the pickle the detector writes inside the container lands where the caller asked for it.
…nd record the resolved CLIP identity The ConceptGraphs floor-noise gate assumed the world floor sits at z=0. Replica's world frame has the floor at z=-1.51 m, so every couch, desk, shelf and cabinet was discarded and only wall-mounted picture frames survived. Add floor_z_m (env SCENE_CG_FLOOR_Z_M, default 0 so Webots is unchanged) and forward it from the launcher. The export recorded the constructor's clip_model_name / clip_pretrained, which are None whenever the default or env value is used, so scoring could not tell which text encoder to pair with the features. Resolve the names once in the detector and export those.
… the manifest The compute budget of a deployment is now stated in scene.config.perception.profile instead of being implied by whichever weights the image bakes in. lite keeps today's model set, full loads SAM-L + CLIP ViT-H-14 from SCENE_MODELS_DIR (mounted at /opt/models/full), and annotate turns object recognition off so boards without a usable GPU still serve regions, annotations and the geometric queries. The perception block is parsed once in the capability module: profile, period_s, confidence_threshold, max_detections and a concept_graphs override table are honoured; any other key is logged as ignored, because earlier manifests carried a large perception block that no code read.
…and text-guided features; deterministic object relations Perception backend (ConceptGraphs detector), following the ThinkGraphs recipe: detections associate to tracks by the share of their voxels the track has observed blended with CLIP text similarity of the labels (association=voxel_vote, the upstream visual+IoU matching stays as association=cg); each track votes on its label with detection confidence; the exported CLIP feature is the bank entry best aligned with the consensus label instead of the running mean of every crop; an area gate keeps distant or occluded crops out of the bank. Side tables live outside the object dicts because upstream's merge rejects unknown keys. Text encoding from the tick thread no longer takes the non-reentrant inference lock. Relations: the geometric loop emits on_top_of/under/inside/contains/near from strict box predicates (footprint and size gates against the old full-volume and same-surface misfires; near capped per object), so the 3D preview shows edges without an LLM. SCENE_RELATIONS=reachable_only restores the previous behaviour.
Scene's metric tier gains a second open-vocabulary mapper: DualMap (Eku127/DualMap, RA-L 2025, Apache-2.0), picked per deployment with `scene.config.perception.backend: dualmap` (env SCENE_PERCEPTION_BACKEND). DualMapDetector subclasses ConceptGraphsDetector so frame fetching, the camera-to-map transform, registry reconciliation, the 3D snapshot and the frame bundle are shared; it replaces model loading (Hydra-composed DualMap config), the per-tick mapping step, text embedding (MobileCLIP-S2) and the Replica export (labels embedded with ViT-B-32 so the upstream scorer runs unchanged). Behaviour learned from live Replica replays and encoded here: - frames pass DualMap's own keyframe rule (0.1 m / 3 deg / 5 s) instead of every tick; feeding every tick gave 100 objects where stride-10 gives 59 - DualMap's merge_local_map() runs every 20 keyframes and the merged object keeps the uid of its dominant constituent so registry ids do not churn - FastSAM follows keep_unknown: its segments only become "unknown" objects, and a CUDA error raised in its thread poisoned one run - 20 consecutive CUDA failures stop the tick loop with one error line - the export is written every 100 ticks and at stop, and an empty map still writes a file so a scorer never reads a stale run - delete_object / reset_derived_state reach DualMap's local map, otherwise the object came back on the next tick Also fixes the 3D web view, which never drew scene-graph edges for any backend: meshes were keyed by backend uuid while edges name registry ids. The snapshot now carries object_id and the page indexes both. docker/Dockerfile.dualmap builds the image (DualMap at 157235e plus its two extra weights) on top of the Scene image; start.sh forwards the new variables and mounts SCENE_DUALMAP_CLASSES. README documents profile, backend, the dualmap knobs and the measured Replica numbers. Validation: system/scene tests (16 in the two touched files, 190 collectable overall; the 11 failures and 5 collection errors are pre-existing and unchanged), seven live Replica room0 replays through the replay primitive (scores n=1 mIoU 12-15 / mAcc 28-32 / F-mIoU 17-20 vs Scene lite 4.6 / 18.3 / 5.4; ReplicaSSG relation R@1 0.32-0.36 vs 0.12), the image built from docker/Dockerfile.dualmap on workpc.
… rate DualMap's lifecycle is sized for a dataset replay that maps every frame: a track needs 8 observations to become stable, and one that leaves the 10 most recent frames without getting there is deleted. A robot mapping keyframes at walking pace sees each object a handful of times, so the map emptied out behind it — one 180 s Webots office run went from 65 tracks to 1, and Scene ended with a single object. stable_num, active_window_size and max_pending_count are now settable per deployment and set for Webots. That alone raised recall from 0.04 to 0.75 and exposed what the empty map had been hiding: 32 false positives, of which 11 were one bin re-registered along the camera ray (0.27 m down to 0.02 m, centres on a line) and the rest tracks of one object whose label flickered between classes. Three export-time filters address them — containment across classes, single-link clustering of same-class tracks whose bounding spheres meet, and a floor and minimum-size gate. False positives fall to 7 (precision 0.36 -> 0.70) while true positives hold at 16. mcp 2.x renamed FastMCP to MCPServer, which breaks scene_service/mcp_tools.py and robonix-api at import; both now require mcp<2.
github-actions
Bot
added
comp:scene
…repo
The restored harness imports numpy and Pillow, which CI did not install, so
the whole module failed to import and its 25 tests were silently skipped
while the job still went red.
Installing them exposed three assertions the harness carried over from the
branch it came from, each describing a state this repository is not in:
- it read `robonix_manifest.mapping-nav-eval.yaml`, which does not exist
here, and expected re-rank configuration in the default deployment
manifest, which leaves `scene: {}` and takes the package defaults. Now
both scene-eval manifests are checked instead, so the DualMap variant
cannot drift from the one it was copied from.
- it expected the Webots camera devices to publish in frames named after
the devices. da97b9c made both publish in the optical frame the depth
registration is expressed in; the assertion follows that, and now also
pins the identity static TFs that bridge the URDF link names.
- it expected `primitive/camera/extrinsics` to appear nowhere. The camera
primitive owns that contract and the deployment binds it; what must not
read a camera topic is the scoring harness, which is what is asserted.
Verified in a clean venv holding only what this job installs: with numpy and Pillow the module then fails on yaml, and the occupancy evaluator needs scipy's KD-tree. All 82 tests pass with the four.
…second
The adapter ran DualMap with `run_local_mapping_only: true` and dropped every
object the local map promoted. That is half the method — the half the name
refers to. The local map associates observations within a class using geometry
and CLIP; a track that becomes stable is promoted to the global map, which
merges across classes by top-down 2D box overlap. Discarding the promotion lost
those objects outright, and left the cross-class merge undone.
So the merge got reimplemented on the export path: containment suppression
across classes, single-link clustering of same-class boxes, a minimum-size
gate. All three are removed here. The global map does that work, upstream, and
`compute_global_spatial_sim` says as much in its own comment ("for global
matching we use geometry only"). Scene now exports the union of both maps;
promotion preserves the uid, so an object does not change identity when it
crosses over.
The floor gate stays — it drops depth noise lying on the floor plane, which is
Scene's own concern and already guards the concept_graphs backend.
Two settings stopped promotion. max_pending_count is not only how long an unstable track survives outside the active window, it is also how long a STABLE track waits before it is promoted, and raising it to 20 (with a 40-frame window) meant no track was promoted in a 180 s office run. Only stable_num needed to move for a robot's observation rate; the window and the pending count go back to DualMap's defaults. The dualmap backend also read its floor height from a private default of 0 m instead of the shared perception setting, so on Replica (floor at -1.51 m) it dropped 21 objects below the world origin as floor noise. It now reads the same setting concept_graphs does, which SCENE_CG_FLOOR_Z_M sets.
An observation joins an existing DualMap track when cos(CLIP) + point overlap exceeds sim_threshold, and overlap counts the points whose nearest track point lies within downsample_voxel_size — 2 cm by default, which assumes the ground-truth poses of a dataset replay. This deployment's SLAM pose is off by 3-7 cm (P95) between keyframes, so two views of one object never overlapped: 60% of detections started a new track, 54 of 57 exported objects had a single observation, none reached stable_num, none was promoted, and every keyframe added another copy. Both knobs are now settable per deployment; the Webots manifest gates overlap at 5 cm.
…the local map DualMap's global map is a navigation memory: only tracks it judges low-mobility (furniture) are promoted into it, and every other stable track is dropped from the local map once it leaves the active window. Running both maps on the Replica replay halved the object count (41 -> 18) and the segmentation score with it. DualMap's own Replica evaluation runs local-only (runner_dataset.yaml), which keeps every stable track; so does this backend now unless a deployment sets perception.dualmap.global_map: true.
The variable was forwarded but the directory was not, so pointing the backend at a DualMap checkout on the host could never work.
On Replica it deleted every rug and carpet (2-5 cm above the floor) and took the rug class from 0.16 IoU to 0, floor with it. The Webots depth error it was written for is absorbed by association now that the overlap gate matches the pose error. Keep it as an opt-in for deployments with real floor noise, and let Webots tracks live long enough to be re-observed.
…se error Keeping tracks alive long enough to be re-observed brought Webots recall back (TP 18, FN 5) and duplicates with it (24). DualMap's local-map self-merge is the upstream answer, but its 0.9 point-overlap threshold assumes ground-truth poses and never fired here (65 -> 65 in every merge log line). The threshold is now settable; Webots relaxes it to 0.3 and merges every 10 keyframes.
Association and the class vote both need an object seen more than once; at 2 s per tick most objects were seen exactly once. The GPU has room now that mapping runs slam_toolbox instead of RTAB-Map's visual odometry.
The scene benchmark's reactive sweep reads lidar sectors and turns at 0.6 m; it exists to move the camera, not to navigate, and it does clip walls. This variant adds the deployment's nav2 service and the explore skill (verbatim from the default manifest) so the same scene and mapping configuration can be evaluated under real navigation (ROBONIX_SCENE_BENCHMARK_MOTION_MODE=explore).
The explore skill's frontier goals sit at wall boundaries and nav2 ends up spinning against them; a demo or a scored run that wants real navigation without that failure can send hand-picked goals through the same MCP surface the skill uses.
The 2D lidar sees one horizontal slice, so a table top is not in the costmap and the robot drives under it into the legs — which is what happened during an explore run. The depth camera is already running for scene; its registered point cloud now also marks obstacles in the local costmap, between 0.08 m and 1.20 m, which covers the table tops, door frames and shelves the scan plane misses. Marking only: a camera frustum is narrow and must not clear what the lidar marked outside it. This is avoidance, not localization — the lidar remains the only input to SLAM, so the map and relocalization are unchanged.
Marking without clearing means every stray point stays an obstacle forever, so the robot dragged a wake of high cost through the costmap. Clearing is per-source raytracing — it only clears along the camera's own rays and cannot erase what the lidar marked elsewhere — so it is safe to turn on. The floor was also being marked at 0.08 m; 0.20 m clears the chassis pitch and depth noise while still catching table tops at ~0.7 m, and the range is trimmed to 2 m where depth error starts marking the floor again regardless.
Marking the depth cloud into the local costmap did not stop the robot hitting tables and left the costmap harder to read, so it goes back out rather than staying in half-working. What the 2D scan plane cannot see remains unsolved; the finding and the measurements stay in the notes.
Drop the benchmark harness (runner, sweep, report renderer, motion and visibility helpers), its 1772-line test and fixtures, the dropout and quality review scripts and their scenarios, the occupancy evaluator, and the CI step that only existed to install that harness's dependencies. What remains is the backend, the two manifests it runs under, the scorer that produced every number in this PR, the ground-truth export, and a label judge that asks a model whether two category names mean the same object -- string equality was calling a monitor labelled 'tv' wrong. The rbnx port-check test travelled here by a merge and belongs in its own change.
Twenty-five tracks labelled `desk` stood for eight tables, and calling that "over-segmentation" hid three unrelated faults that need three different cures. Nine were the floor. It arrives through the detector as a slab 1.6 mm thick carrying two thousand points -- more evidence than any real table -- so every filter that ranks by evidence keeps it and drops the furniture instead. Being flat is not enough to reject on (a keyboard is flat, measured 7-13 mm, and a rug is 20-50 mm); being flat AND lying on the floor is, because nothing the robot is asked to find is. Nine were fragments 0.11 m across still labelled `desk`. The scale to judge them by comes from the class's own members rather than a table of expected sizes: a vocabulary is the deployment's own list and nobody should have to write down how big a desk is for their site. A track has to be both four times smaller and ten times sparser than the largest of its class, so a class holding one large and one small member keeps both, and a class with fewer than three members is left alone for want of a scale. The rest were one table seen from two sides. Same-class absorption now fires when either box claims the other's space -- half the smaller box inside the intersection, or the smaller centre inside the bigger box. Neither test alone is enough: overlap misses a long thin smear copy whose centre is well inside the original (replacing containment with overlap took duplicates 7 -> 16 on the same tour), and containment misses a table met in halves, whose pieces sit beside each other rather than nested. Two tables pushed together satisfy neither. Measured offline on a recorded office run: 88 tracks -> 49, duplicates 24 -> 8, ghosts 42 -> 21, F1 0.336 -> 0.493, at the cost of one true positive.
Three things the numbers in this PR depend on were not in it. The office vocabulary lived in a throwaway script that edited the manifest in place, so the run that took label accuracy from 0.43 to 0.74 could not be repeated from a checkout -- and the manifest it edited has since been restored, which means the recent rounds silently ran DualMap's 101-name home list again (`wardrobe` and `bed` among the ghosts, in an office). Both DualMap manifests now carry the 40 names. The benchmark file the scorer reads was dropped along with the harness when this PR was slimmed down. `evaluate_webots_scene.py` defaults to it and cannot run without it, so the scorer that produced every Webots number here did not work in the tree that ships it. And there was no ConceptGraphs counterpart to the DualMap manifest, so "before and after" had no before: every Webots row compared DualMap against itself. `scene-eval.cg.nav.yaml` is the same world, the same saved map, the same nav2 tour and the same scorer, differing only in the backend the manifest selects.
…nds by the map The status page now names the perception backend and shows what it is costing: the scene container's CPU and RSS, the GPU memory this process has reserved, the card's total and utilisation, and the perception loop's last tick -- four rolling curves at 1 Hz on its own panel (/cost, /api/cost). A backend chosen for being light has to be able to show it, and until now the only way to learn which backend was running was to read the manifest. The occupancy-consistency gate -- an object cannot stand on ground the robot has never observed -- moves out of the DualMap backend into a module both backends use. It is a statement about the world, not about a detector, and living in one backend meant the other shipped without it: the concept_graphs rows registered objects outside the walls. concept_graphs objects carry a centre and a box rather than a cloud, so the gate samples the footprint. Each backend also records how long one pass takes, timed around the whole tick so a pass that raises is still counted.
A 3.5 x 4.5 m "desk" with 3765 points survived every gate: not thin (the floor plus a few table legs spans 30 cm), the largest of its class (so no scale to call it an outlier against), overlapping nothing of its own class. Its median point sat 5 cm below the floor. Nothing the robot is asked to find has most of itself under the ground it stands on, so that is now a reason on its own; a rug's median is above the floor and stays. The office manifests also turn on the existing floor_gate: it was left off because it deleted Replica's rugs, and an office has none.
enkerewpo
marked this pull request as draft
September 9, 2026 17:23
The backend a deployment gets when it asks for none was a constant, and a constant cannot describe a choice that depends on the image it runs in. DualMap arrives as a layer on top of the scene image, so an image either carries its checkout and weights or it does not, and there is no DualMap build for Jetson at all. A hard default of dualmap leaves every plain image failing at boot on a backend it cannot import; a hard default of concept_graphs leaves the better mapper switched off on the images built to carry it, which is the whole reason those images exist. So the default is resolved from what is present. default_backend() prefers DualMap when the checkout is at /opt/dualmap and falls back to ConceptGraphs when it is not, and start.sh picks the matching image the same way. Building stays explicit: build.sh still has to be asked for the dualmap image, because a default build should not silently become the heavy one. The silent-default test now says which image it is describing rather than leaving it to whatever the machine running the tests happens to have under /opt, which is how it would pass in CI and surprise someone on a workstation that had built the layer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scene's object accuracy had stopped improving, and the honest reason was that its perception was ours: a hand-rolled association loop we kept tuning, with no external reference to say what level it was at. This adds a reproducible published system as a selectable backend, and — for the first time — runs the two side by side on the same world, map, tour and scorer. The result is not the one this PR set out to show: on object F1 the two are level once the map is right, DualMap is clearly better at naming things, and the largest single gain came from the vocabulary rather than from the backend.
What changed
perception.backend: concept_graphs | dualmap. DualMap (Eku127/DualMap, RA-L 2025, Apache-2.0, pinned at 157235e) runs as a Scene detector: YOLO-World + MobileSAM/FastSAM + MobileCLIP-S2, its own keyframe rule, its own local map and its own self-merge. Nothing above the detector changes — the object registry, the geometric relation loop, the capability surface and the web views are the same.concept_graphsstays the default.The DualMap object lifecycle is a deployment setting, and this was a real defect. DualMap's lifecycle is sized for a dataset replay that maps every frame: a track needs
stable_num(8) observations to become stable, and one that leaves theactive_window_size(10) most recent frames without getting there is deleted aftermax_pending_countrounds. A robot mapping keyframes at walking pace sees each object a handful of times, so the map emptied out behind it — one 180 s Webots office run went from 65 tracks to 1, and Scene ended with a single object. Recall went from 0.04 to 0.75 once these matched the observation rate.The vocabulary is a deployment setting too, and it is the largest lever in this PR. DualMap ships a 101-name list built for homes — bed, bathtub, ironing board. A detector can only answer with a name it was given, so in an office it called the cabinet a sink and the chair an ironing board: of the 38 objects the benchmark counted as invented, 27 were within a metre of a real one. Right place, wrong name.
perception.dualmap.classestakes the deployment's own list; the office manifests carry a 40-name office vocabulary, deliberately wider than what the world contains so the score still measures recognition rather than a lookup. Label accuracy went from 0.43 to 0.74 on that change alone.Labels are judged for meaning, not spelled the same. The scorer matches by geometry and then compares labels; string equality called a monitor labelled
tv, a couch labelledsofaand a plant labelledvasewrong, which is a statement about vocabulary rather than recognition.testing/judge_scene_labels.pyasks a model whether each (expected, observed) pair names the same kind of object and writes the judgments fileevaluate_webots_scene.pyalready accepts. It is strict on purpose — cabinet/shelf, keyboard/monitor and window/mirror are distinct — and on the office run it found exactly one equivalence (couch/sofa): with the right vocabulary, the remaining label errors are genuine confusions.Two merges that were tried and taken back out. Both were written to attack duplicate objects, both reduced duplicates, and both were removed because they traded them for something worse:
global_map: true) and a wider association voxel. Duplicates fell (31 → 6) while true positives fell with them (40 → 24) and ghosts rose — distinct nearby objects were being folded into one. Both stay available as opt-in settings; neither is on.mcp<2. mcp 2.x renamed
FastMCPtoMCPServer;scene_service/mcp_tools.pyandrobonix-apiuse the v1 API and now fail at import without the pin.sim/stop.shno longer kills processes it does not own. It matchedrobonix-atlas|robonix-executor|...across the whole machine, so running it took down an unrelated Robonix deployment on the same host. It now scopes by working directory and cmdline ancestry under the Webots checkout.Replica (ConceptGraphs scorer, live replay through Scene)
For reference on the same scorer: DualMap standalone with its general vocabulary is 13.1 / 27.6 / 19.9 at n=1, and the offline paper-recipe ConceptGraphs (SAM ViT-H + CLIP ViT-H-14) reaches 39.8 / 21.4 mAcc / F-mIoU across 4 scenes. Scene's live numbers track DualMap standalone, which is the point — the backend is reproduced, not reinvented.
What this comparison is, and is not
The in-repo backend runs its
liteprofile in every row below — YOLO-World detection, MobileSAM segmentation, CLIP ViT-B-32 features — not the ConceptGraphs paper's configuration, which segments whole frames with SAM ViT-H and encodes with CLIP ViT-H-14, offline. Afullprofile with the larger weights exists in the repository, but a 6 GiB card cannot hold it alongside the simulator, and offline batch processing is not how a robot runs.So these rows compare our lightweight online loop against DualMap's default configuration, on the hardware this deployment uses. That answers whether switching backends is worth it here. It does not establish which underlying method is stronger, and the effect of the smaller segmentation and feature models on the in-repo backend's numbers has not been measured — ruling it out would need a third arm running the
fullprofile on a larger card.How the Webots rows are measured
The comparison is only worth reading if the ruler is stated, so:
Ground truth comes out of the world file, not off a clipboard.
export_webots_scene_truth.pyreadsoffice.wbtand turns its nodes into truth objects through the 62 semantic types intesting/fixtures/webots_scene_benchmark.json(Sofa → couch, 2.0 x 0.9 x 0.8 m, association radius 0.75 m, and so on). Change the world and the truth follows.Only what the robot actually saw is scored. An RGB-D sweep produces
visible_truth_ids; a truth never in view is not counted against recall, or the score measures the tour rather than the perception. The office tour covers 1.0 (all 51 truths seen), so recall here is clean.Association is geometric, deliberately blind to the label. A Hungarian one-to-one assignment over hard admissibility gates (a per-type ground-plane radius, 0.5–0.9 m; bbox volume ratio >= 0.03) maximizes match count, then prefers semantically correct labels, then minimizes normalized XY distance. Keeping the label out of association is the point: a detection in the right place with the wrong class must count as a label error, not as a miss.
False positives are split, because they need different cures.
Merging fixes duplicates; filtering fixes ghosts. A single FP number hides which of the two a change moved, which is exactly what went wrong in the rounds below.
Labels are judged for meaning (see the judge script above), and the gates in the benchmark are precision >= 0.75, recall >= 0.7, F1 >= 0.72, label accuracy >= 0.8, duplicate rate <= 0.1, ghost rate <= 0.05. None of them passes yet.
A round only counts when localization stayed healthy. Scene can only register what the robot drove past. One round scored F1 0.342 off 25 objects; the cause was a localization the mapping service had marked
suspect, four of five nav2 goals failing, and a robot that barely moved — a navigation failure being read as a perception result. Rounds now require 4 of 5 goals to succeed with at most onesuspectreading during the tour, and are discarded otherwise. The two backends are run alternately, not one after the other, so any drift in the machine lands on both sides equally.Webots office: the backend swapped, nothing else
Every row: the office world, the same saved map, the same five nav2 goals, the same scorer, one image (
robonix-scene-dualmap, which carries both backends' weights), and onlyperception.backendchanged in the manifest. A round counts only if 4 of 5 goals succeeded with at most onesuspectlocalization reading; invalid rounds were retried, not reported.Before two fixes landed the same evening — the saved map's
free_threshserved unexplored cells to nav2 as free floor (syswonder/service-map-rbnx#24), and tour goals were picked with a nine-point clearance check that let a 0.25 m gap through. Both backends ran on that map, so the rows are comparable with each other and not with the group below.concept_graphs(Scene as on dev)dualmap, DualMap's own 101-name home listdualmap+ the 40-name office listAfter (unknown cells stay unknown; goals need 0.9 m of real clearance):
concept_graphsdualmap+ office listWhat the two tables say, stated plainly:
concept_graphs0.742 / 0.584 with a 0.16 swing between rounds,dualmap0.707 on one round. This PR does not claim DualMap scores higher on object F1. Three rounds a side would be needed to say either way.What each backend costs
Not measured until this evening, and the reason DualMap was chosen in the first place. Measured on the workpc (RTX 2060, 6 GB; 30.6 GB host RAM), sampled every 5 s on the same tour. "RSS" is the scene container's memory as
docker statsreports it — host RAM, not GPU memory; GPU memory is its own column, and both backends fit the 6 GB card.concept_graphsdualmap+ office listThe steady state differs modestly and traceably: DualMap voxel-downsamples an object's cloud after every fusion, while the in-repo backend does so only once an object passes a per-object cap, so the container settles at 2.3 GB against 2.8 GB. CPU differs more, and also traceably — the in-repo backend runs detection, segmentation and encoding every tick; DualMap runs them only after 0.15 m of travel or 5 degrees of turn.
Separately, and worth reporting because of what it would mean on a robot: on one tour the in-repo backend's resident set went from 2.8 GB to 22.0 GB within one sampling interval, stayed there two minutes, then dropped to 0.9 GB as the container restarted. DualMap showed nothing comparable on the same route. This is an excursion, not gradual growth, and it is not being called a leak: the cause is not localized yet (the periodic merge pass, which concatenates two clouds and re-processes them through an optional DBSCAN, is the leading suspicion). Repeated rounds are running to establish whether it reproduces.
The scene status page now shows these live, with the backend's name (
/cost).Validation
system/scene/tests/test_perception_dualmap.py— map-entry shaping, the manifest vocabulary, keyframe gating, uid preservation across merges, and the lifecycle plumbing.system/scene/tests/test_capabilities.py— backend routing and config validation (an unknown key underperception.dualmapfails at load).Notes
docker/Dockerfile.dualmap) clones the pinned commit and downloads FastSAM-s and MobileCLIP-S2 with size checks; Scene's own code is still bind-mounted.examples/webots/robonix_manifest.scene-eval.dualmap.nav.yamlis the manifest these runs used; the.dualmap.yamlvariant is the same without nav2.Screenshot
Scene's live view on the DualMap backend in the Webots office world. Left is the 2D map with tracked objects, centre the 3D view with object boxes and relation edges, right the live RGB and depth the backend is reading.
