From 307543adbd3b672980d246a71dd5ef3041001e50 Mon Sep 17 00:00:00 2001 From: alex Date: Tue, 21 Apr 2026 15:40:48 +0200 Subject: [PATCH 01/25] Started prototyping. Got a recolor event working in the notebook. Need to convert to a variation. --- 2026_04_13_sensitivity_analysis.md | 611 ++++++++++++++++++ 2026_04_21_color_variation_status.md | 39 ++ 2026_04_21_variation_system_plan.md | 142 ++++ .../examples/compile_env_notebook.py | 88 ++- isaaclab_arena/examples/tint_events.py | 173 +++++ 5 files changed, 1049 insertions(+), 4 deletions(-) create mode 100644 2026_04_13_sensitivity_analysis.md create mode 100644 2026_04_21_color_variation_status.md create mode 100644 2026_04_21_variation_system_plan.md create mode 100644 isaaclab_arena/examples/tint_events.py diff --git a/2026_04_13_sensitivity_analysis.md b/2026_04_13_sensitivity_analysis.md new file mode 100644 index 000000000..6c5d2e1a9 --- /dev/null +++ b/2026_04_13_sensitivity_analysis.md @@ -0,0 +1,611 @@ +# Sensitivity Analysis \- Motivation, Goals, and Design + +This document is intended to lay out: + +* The **goals** of the “Sensitivity Analysis” part of Isaac Lab Arena v0.3 +* What pieces are **missing** that we’re required to build to achieve those goals, and +* What **exists** out there that serves as motivation? + +# Goals and Motivation + +# The main goal of the "Sensitivity Analysis" goal in the v0.3 POR is to enable systematic analysis of how factors (environmental/robot) affect VLA policy performance, by using controlled perturbations within the Isaac Lab Arena. + +# When evaluating a VLA policy across a multi-task benchmark, a binary success rate tells you how well the policy performs, but not why it succeeds or fails. Two policies with the same 30% success rate may fail for entirely different reasons: one may be brittle to camera placement, another to lighting, another to object pose. Without a way to systematically vary these factors and measure their impact, we cannot diagnose failure modes or guide improvements to either the policy or the evaluation setup. + +# The [RoboLab](https://research.nvidia.com/labs/srl/projects/robolab/) project (an internal research benchmark) demonstrated the value of solving this issue. Their sensitivity analysis, published in their paper (NVIDIA, 2026), uses Bayesian posterior estimation (MNPE) over evaluation data to answer questions like "How does policy performance degrade with camera miscalibration?" and "Which lighting conditions are most associated with failure?". Example findings included: + +* **Wrist camera** pose was the single strongest predictor of success across all policies tested, with the posterior sharply concentrated near zero displacement. Policies were far more tolerant to displacement of the external camera. +* **Object distance** from the robot showed a clear peak at \~0.5m, corresponding to the robot's reachable workspace. + +These insights are exactly what Arena users need: understanding not just overall success rates, but which scene parameters drive that performance. **Such insights are *actionable* \- they tell a user what should be changed in order to make the policy better.** + +# What's missing in IsaacLab-Arena today? + +Arena currently has the building blocks for running individual evaluations, but lacks the infrastructure to run controlled perturbation studies and analyze the results. Specifically, we will address two key missing pieces: + +1) **Variation System**: We need a unified system for varying environment parameters across evaluation runs in IsaacLab-Arena + +2) **Analysis Tooling**: Provide dedicated sensitivity analysis tooling (eg. Bayesian posterior estimation \- Take evaluation results and determine which parameters matter.) + +# High-Level API + +This section describes ***what we want*** without considering how we might implement it. + +At a high level, we want a user to be able to: + +* Evaluate their policy in simulated environments with variations selected via CLI arguments/configuration files (rather than having them hard-coded in a script), +* Generate a report that details the sensitivity of output metrics to input variations. + +**Run the policy in an environment with variations** + +Running a single policy on a pick and place task, through our policy\_runner.py script, with variations over the pick\_up\_object, destination object, and lighting could be described as: + +``` +python policy_runner.py \ + --num_envs 10 + franka_srl_pick_and_place \ + pick_up_object.name="Choose(apple, banana, pear)" + pick_up_object.mass="Uniform(0.1, 1.0)" + pick_up_object.color="Choose(red, blue, green)" + destination_object.name="Choose(bowl, pot)" + light.color="Choose(red, blue, green)" +``` + +Running a multi-task, multi-policy test using an experiment configuration file. + +``` +# Run +python isaaclab_arena/evaluation/eval_runner.py \ + --viz kit \ + --eval_jobs_config isaaclab_arena_environments/eval_job_configs/franka_pnp_and_open_door_sensitivity.yaml +``` + +With experiment config file: + +``` +# isaaclab_arena_environments/eval_job_configs/franka_pnp_and_open_door_sensitivity.yaml + +policies: [groot_n16, pi_05] + +pick_and_place_srl_table: + embodiment: franka + pick_up_object: + name: [apple, banana, pear] + mass: Uniform(0.2, 0.5) + light: + color: [red, green, blue] + intensity: Normal(1000, 100) + +open_door_kitchen: + embodiment: franka + pick_up_object: + name: [microwave, toaster_oven] + joint_stiffness: Normal(1.0, 0.1) + light: + hdr: [billiard_hall_robolab, home_office_robolab] +``` + +**Analysing the results (sensitivity analysis)** + +Run an analysis of the output results. This will output a plot detailing the sensitivity of one output metric to one input (variation) factor. + +``` +python analyze_sensitivity.py \ + --results_dir output/franka_pick_and_place_experiment \ + --input_factor pick_up_object.mass \ + --output_metric success_rate \ + --figure_path my_plot.png +``` + +It’s likely that a user will want some interactive way of inspecting the sensitivity of the policy. In this website the user could click on some output metric, and then click on some input factor and get some plot of the sensitivity. + +``` +python sensitivity_website.py \ + --results_dir output/franka_pick_and_place_experiment +``` + +These metrics provide a high-level view of a policy’s performance and its sensitivity to various factors. **This information indicates avenues for improving policy performance \- the whole point of Arena\!** + +# Existing Works **TLDR** + +This short section summarizes the analysis of other frameworks. For the full analysis, including code examples, see: [Background](#existing-works-full). + +**Analysis Tooling** + +* **Sensitivity analysis:** Robolab does a great job of this; we should take this verbatim. +* **Subtasks and Metrics:** Robolab does a great job of this, and we should close the gap between our sub-tasking/metrics system and theirs. + +**Variation System** + +* **Modular Variations:** Colosseum and FactorWorld have a modular system for adding variations that we should take inspiration from. +* **Variation configuration:** FactorWorld allows all parameters of all sources of variation to be configured on the CLI through Hydra. When adding new variations, no additional work is required to have their parameters appear on the command line. Maximum configurability \- we want this. +* **Variation Implementation:** In Colosseum and FactorWorld, each variation is represented as a class that can be registered with a central variations system. We want this extensibility and modularity. +* **Experiment specification:** No reviewed system has a great way of specifying experiments through an experiment specification file. We can make our own improvement here (see above for proposal). + +# Existing Works **FULL** {#existing-works-full} + +This section covers relevant details on how others perform sensitivity analysis. The idea is to take inspiration from the good points and see how we could improve. + +## **RoboLab** + +This section covers how Robolab specifies variations. Sensitivity analysis, i.e. how you make sense of the results *after* the variations have been applied, is done via Simulation-Based Inference (SBI), which is a separate topic. + +Robolab has two mechanisms for specifying variations: + +* **Registration Time:** These bake variations in the env\_cfg by generating several environments by varying factors, say the lights, which are baked in the env\_cfg. + +* **Run-time:** These are randomizations that occur in each environment when it is run. For example, the camera extrinsics randomization, which are implemented as event terms. + +Specifying and applying these variations are handled by top-level experiment scripts (e.g. \`run\_eval\_camera\_pose\_variation.py\` and \`run\_eval\_lighting.py\`). This top-level script takes charge of registering environments (including variations) and inserting randomization events into the environments. + +**Interface** +Variations are implemented through top-level scripts. There is 1 script per type of variation. For example: + +``` +bash + +# Runs the cartesian product of all environmental variations and camrea pose variation types +python robolab/examples/run_eval_camera_pose_variation.py +``` + +**Implementation** +Variations are hardcoded in these top-level scripts: + +``` +robolab/examples/run_eval_camera_pose_variation.py + +# Define perturbations as constants +CAMERA_NAMES_EXTERNAL = ["external_cam"] +CAMERA_POSE_RANGE_EXTERNAL = { + "x": (-0.2, 0.2), + "y": (-0.2, 0.2), + "z": (-0.1, 0.1), + "roll": (-0.2, 0.2), + "pitch": (-0.2, 0.2), + "yaw": (-0.2, 0.2), +} + +... + +# Register environment, with the purturbation in. +env, env_cfg = create_env(base_task_env, + device=args_cli.device, + num_envs=num_envs, + use_fabric=True, + events=camera_pose_event, # Here is the purturbation + policy=args_cli.policy) + +# Run the eval +env_results, msgs, timing = run_episode(env=env, + env_cfg=env_cfg, + episode=run_idx, + save_videos=args_cli.save_videos, + headless=args_cli.headless, + remote_host=args_cli.remote_host, + remote_port=args_cli.remote_port) +``` + +**Supported Variations** +What variations does this framework support? + +* **manipulated object:** \- *(Through USDs not configuration)* +* **receiver object:** \- *(Through USDs not configuration)* +* **background:** background\_hdr, lighting, table material +* **physical:** \- +* **robot:** camera\_pose + +**Pros and Cons** +What can we learn from this framework? + +* **Pros** + * It works and produces amazing results (see paper) + * **Sensitivity Analysis:** They have a mature framework for measuring the sensitivity of the policy to varied factors. + * **Subtasks \+ Metrics:** Xuning has a very mature sub-task/event system that tracks progress towards the ultimate goal (task success). This generates a rich output dataset, against which the inputs (environmental variation) can be correlated. + * **Tasks variance:** Unlike the frameworks below Robolab supports more general tasks than a single table top pick and place with a single focus object. +* **Cons** + * **No variation configuration system:** No unified way of specifying an experiment that includes a list of arbitrary variations. Experiments are effectively captured in the top-level scripts and they test one, hard-coded source of variation. We want a user to be able to specify variations more generally, in some experiment configuration file, without rewriting a top-level script. + * **Output format:** Hard-coded output summary per experiment type. We want some automatic output based on the user specification of the input variations. Basically, without the user having to write some top-level experiment script and specify the data to be saved to the output, all data required for all potential analyses (given the input variations, and output metrics) should be saved. + +![][image1] +Figure 1.0 \- An example of a sensitivity analysis performed in Robolab, of the success rate with respect to the extrinsic camera calibration. + +## **The Colluseum** + +**Interface** +Each environment gets a file describing the possible variations + +``` +colosseum/assets/configs/basketball_in_hoop.py + +env: + task_name: "basketball_in_hoop" + seed: 42 + scene: + factors: + + - variation: object_color + name: manip_obj_color + enabled: False + targets: [ball] + seed: ${env.seed} + + - variation: object_color + name: recv_obj_color + enabled: False + targets: [basket_ball_hoop_visual] + seed: ${env.seed} +``` + +Then we have another file describing experiments to be run + +``` +colosseum/assets/json/basketball_in_hoop.json + +{ + "strategy": [ + { + "spreadsheet_idx": 1, + "variation_name" : "all_mixed", + "enabled": true, + "variations": [ + {"type": "object_color", "name": "manip_obj_color", "enabled": true}, + {"type": "object_color", "name": "recv_obj_color", "enabled": true}, + {"type": "object_texture", "name": "manip_obj_tex", "enabled": true}, + {"type": "object_texture", "name": "recv_obj_tex", "enabled": true}, + {"type": "object_size", "name": "manip_obj_size", "enabled": true}, + {"type": "object_size", "name": "recv_obj_size", "enabled": true}, + {"type": "light_color", "name": "any", "enabled": true}, + {"type": "table_color", "name": "any", "enabled": true}, + {"type": "table_texture", "name": "any", "enabled": true}, + {"type": "distractor_object", "name": "any", "enabled": true}, + {"type": "background_texture", "name": "any", "enabled": true}, + {"type": "camera_pose", "name": "any", "enabled": true}, + {"type": "object_friction", "name": "any", "enabled": true}, + {"type": "object_mass", "name": "any", "enabled": true} + ] + }, + { + "spreadsheet_idx": 2, + "variation_name" : "manip_obj_color", + "enabled": true, + "variations": [ + {"type": "object_color", "name": "manip_obj_color", "enabled": true}, + {"type": "object_color", "name": "recv_obj_color", "enabled": false}, + {"type": "object_texture", "name": "manip_obj_tex", "enabled": false}, + {"type": "object_texture", "name": "recv_obj_tex", "enabled": false}, + {"type": "object_size", "name": "manip_obj_size", "enabled": false}, + {"type": "object_size", "name": "recv_obj_size", "enabled": false}, + {"type": "light_color", "name": "any", "enabled": false}, + {"type": "table_color", "name": "any", "enabled": false}, + {"type": "table_texture", "name": "any", "enabled": false}, + {"type": "distractor_object", "name": "any", "enabled": false}, + {"type": "background_texture", "name": "any", "enabled": false}, + {"type": "camera_pose", "name": "any", "enabled": false}, + {"type": "object_friction", "name": "any", "enabled": false}, + {"type": "object_mass", "name": "any", "enabled": false} + ] + }, +``` + +which sets variations on and off per experiment. + +**Implementation** +The colosseum exposes variations through classes inheriting from \`IVariation\` class for example: + +``` +colosseum/variations/object_color.py + +class ObjectColorVariation(IVariation): + """Object color variation, can change objects' color in the simulation""" + + def __init__( + self, + pyrep: PyRep, + name: Optional[str], + targets_names: List[str] + ): + ... + + def randomize(self) -> None: + """ + Samples a random color and sets it to the objects in the simulation. + Depending on the self._color_scame parameter, all objects will receive + the same color or different colors otherwise if the parameter is false + """ + ... +``` + +Internally, when “randomize” is called this object looks up the objects specified through “target\_names” and changes the colors. + +**Supported Variations** +What variations does this framework support? + +* **manipulated object:** color, texture, size +* **receiver object:** color, texture, size +* **background:** light\_color, table\_texture, distractors, background\_texture +* **physical:** object\_friction, object\_mass +* **robot:** camera\_pose + +**Pros and Cons** +What can we learn from this framework? + +**Pros:** + +* **Config files:** Configuration files describe the possible variations in a scene and what variations are run in an experiment. +* **Variations as objects:** Variations are modular, their code is isolated in their own classes, and they can be registered to tasks/environments (rather than harded coded in some experiment script at the top level). + +**Cons:** + +* **Coupling between task and experiments:** 1:1 linking between a task/environment “basketball\_in\_hoop” and an experiment. *We* want many different types of experiments (potentially) to run on a single environment. +* **Experiment’s file is verbose:** Having to list all possible variations per experiment run results in verbose json files + +**Summary:** I think this is more in the direction of what we want when compared with RoboLab. Variations are modular and can be registered to tasks. On the other hand, there are improvements that we can make regarding the experiment description simplicity (not requiring every variation to be listed), and the experiment flexibility (removing the 1:1 mapping between experiments and tasks). + +## **Robotwin** + +**Similar to frameworks above. Short section.** This section covers how Robotwin specifies variations. + +**Interface** +Variations are specified in a yaml file, which are also overridable on the CLI. + +``` +bash + +python script/eval_policy.py \ + --config ./task_config/demo_clean.yml \ + --overrides --domain_randomization "{'random_background': True, 'cluttered_table': False, 'clean_background_rate': 0.5, 'random_head_camera_dis': 0, 'random_table_height': 0.03, 'random_light': True, 'crazy_random_light_rate': 0.02}" + +``` + +With a configuration file like: + +``` +robotwin/task_config/demo_randomized.yml + +# Define perturbations as constants +render_freq: 0 +episode_num: 50 +use_seed: false +save_freq: 15 +embodiment: [aloha-agilex] +language_num: 100 +domain_randomization: + random_background: true + cluttered_table: true + clean_background_rate: 0.02 + random_head_camera_dis: 0 + random_table_height: 0.03 + random_light: true + crazy_random_light_rate: 0.02 +camera: + head_camera_type: D435 +... +``` + +**Implementation** +Variations take place in the \_base\_task.py + +``` +robotwin/envs/_base_task.py + +# Define perturbations as constants +def setup_scene(self): + + ... + + direction_lights = kwargs.get("direction_lights", [[[0, 0.5, -1], [0.5, 0.5, 0.5]]]) + self.direction_light_lst = [] + for direction_light in direction_lights: + if self.random_light: + direction_light[1] = [ + np.random.rand(), + np.random.rand(), + np.random.rand(), + ] + self.direction_light_lst.append( + self.scene.add_directional_light(direction_light[0], direction_light[1], shadow=shadow)) + + ... +``` + +So the key thing is “if random\_light:” then randomize the lights. So the randomizations are harded in the base class for all tasks. There’s no registration system. + +A lot of this is possible because ALL tasks occur on a predefined table top. So all randomizations, apply to all tasks. + +**Supported Variations** +What variations does this framework support? + +* **manipulated object:** \- +* **receiver object:** \- +* **background:** clutter, background textures, tabletop height, lighting +* **physical:** \- +* **robot:** \- + +**Pros and Cons** +What can we learn from this framework? + +* **Pros** + * **Config files:** The variations are somewhat configurable through a file. +* **Cons** + * **Variations not extensible/module:** All possible variations are hardcoded in the \_base\_task.py. A user would have to make code changes to base\_task to add a variation. + * **Specific to single tabletop:** The whole framework is centered around a single table top. For example a randomization is the global tabletop height. Object placement is tabletop specific. + +## **Factor World** + +**Similar to frameworks above. Short section.** This section covers how Robotwin specifies variations. + +**Interface** +Variations are specified in a yaml file, which are also overridable on the CLI. + +``` +bash + +python -m run_scripted_policy \ + mode=save_video \ + output_dir=/tmp/data \ + num_episodes=10 \ + num_episodes_per_randomize=1 \ + seed=0 \ + factors=[arm_pos,light,object_size] \ + task_name=basketball-v2 + +``` + +With a configuration file looks like: + +``` +factor-world/cfgs/data.yaml + + # Used to sample factor values for data generation. + factors: + # ----- Default factors ----- # + arm_pos: + x_range: [-0.5, 0.5] + y_range: [-0.2, 0.4] + z_range: [-0.15, 0.1] + num_resets_per_randomize: default + seed: ${seed} + object_pos: + x_range: [-0.3, 0.3] + y_range: [-0.1, 0.2] + z_range: [-0, 0] + theta_range: [0, 6.2831853] + num_resets_per_randomize: 1 + seed: ${seed} +... +``` + +Each factor has it’s own parameters. + +**Implementation** +Each factor has its own class. + +``` +robotwin/envs/_base_task.py + +class ObjectPosWrapper(FactorWrapper): + + def __init__(self, + env: gym.Env, + x_range: Tuple[float, float] = (-0.3, 0.3), + y_range: Tuple[float, float] = (-0.1, 0.2), + z_range: Tuple[float, float] = (-0, 0), + theta_range: Tuple[float, float] = (0, 2 * np.pi), + seed: int = None, + **kwargs): + """Creates a new wrapper.""" + super().__init__( + env, + factor_space=spaces.Box( + low=np.array([x_range[0], y_range[0], z_range[0], theta_range[0]]), + high=np.array( + [x_range[1], y_range[1], z_range[1], theta_range[1]]), + dtype=np.float32, + seed=seed), + ) + + ... + + def reset(self): + super().reset() + + # Reset object pos. + self._set_object_pos( + self.object_init_pos, + self.object_init_quat) + + ... +``` + +One cool thing is that all of these constructor arguments are available through the CLI, through hydra. + +**Supported Variations** +What variations does this framework support? + +* **manipulated object:** object\_pos, object\_size, object\_texture, +* **receiver object:** \- +* **background:** floor\_texture, table\_texture, table\_pos, distractor\_xml, distractors +* **physical:** camera\_pos, light +* **robot:** arm pos + +**Pros and Cons** +What can we learn from this framework? + +* **Pros** + * **Extensibility:** Each factor can be added as a new class and it’s arguments are automatically available through the CLI, through hydra. + * **CLI \+ Hydra:** All the variation factor control are available through the CLI automatically. + * **Config files:** The variations are configurable through a file. +* **Cons** + * **Specific to single tabletop:** The whole framework is centered around a single table top. For example the variations rely on their being a single central object for the task. + +# What are we missing to achieve this? + +TBD + +| Description | Risk | Time Required | Priority | +| ----- | ----- | ----- | :---: | +| | | | | +| | | | | + +# Open Questions + +Here are a list of open questions that I can think of that we need to decide on: + +* **Heterogeneous vs. Homogeneous variations** + * **Question:** How do we deal with the fact that some variations can be run in parallel environments (object position, camera extrinsics, etc.) while some variations need to be run sequentially (for example lighting). + * **Approach** + * Experiment to see what, of the common factors of variation, can be run in parallel, and what can’t. + * Need to come up with an orchestration system where the requested variations are broken down into a series of parallal and sequential runs (or in parallel nodes on OSMO). + +* **Sensitivity website/report** + * **Question:** Is the sensitivity analysis fast enough that a meaningful report/website which updates to user requests is possible? + * **Approach:** prototype. + +# Appendix + +``` +# Run +python isaaclab_arena/evaluation/eval_runner.py \ + --viz kit \ + --eval_jobs_config isaaclab_arena_environments/eval_job_configs/franka_pnp_and_open_door_sensitivity.json + + + +# isaaclab_arena_environments/eval_job_configs/franka_pnp_and_open_door_sensitivity.json + + + + +{ + "jobs": [ + { + "name": "franka_pick_and_place", + "arena_env_args": { + "environment": "pick_and_place", + "embodiment": "franka", + "pick_up_object.name": "Choose(apple, banana, pear)", + "pick_up_object.mass": "Uniform(0.1, 0.5)", + "hdr": "Choose(billiard_hall_robolab, home_office_robolab)", + }, + "num_episodes": 100, + }, + { + "name": "franka_open_door", + "arena_env_args": { + "environment": "open_door", + "embodiment": "franka", + "object_with_door.name": "Choose(microwave, toaster_oven)", + "object_with_door.joint_stiffness": "Normal(1.0, 0.1)", + "hdr": "Choose(billiard_hall_robolab, home_office_robolab)", + }, + "num_episodes": 100, + ], + # Run all of the above tests with the following policies. + "policies": [ + "groot", + "pi05" + ] +} +``` + +[image1]: diff --git a/2026_04_21_color_variation_status.md b/2026_04_21_color_variation_status.md new file mode 100644 index 000000000..34d55a7ee --- /dev/null +++ b/2026_04_21_color_variation_status.md @@ -0,0 +1,39 @@ +# Color Variation — POC Status + +Companion to [2026_04_21_variation_system_plan.md](2026_04_21_variation_system_plan.md). Tracks the state of the per-env object color randomization exploration. + +## Where we are + +We have a working proof-of-concept for per-env color randomization on scene objects, validated in `isaaclab_arena/examples/compile_env_notebook.py` with `num_envs=4`, the kitchen background, and two YCB objects (`cracker_box`, `tomato_soup_can`). + +**What works** — the `mdp.randomize_visual_color` event from Isaac Lab, injected post-`compose_manager_cfg()` on `env_cfg.events`. Each cloned env gets a distinct, random flat color bound to the object's top-level prim. Requires `scene.replicate_physics=False` (Arena default). + +**What doesn't (yet) work** — the in-place diffuse tint variant (`isaaclab_arena/examples/tint_events.py`, class `randomize_visual_diffuse_tint`). It walks the stage, finds each env's bound material shader, and writes `inputs:diffuse_tint` (MDL/OmniPBR) or `inputs:diffuseColor` (UsdPreviewSurface). The event runs without errors but the rendered objects do not visibly change color. Leaving the in-place tint code in the notebook (alongside the commented-out `randomize_visual_color` block) so we can A/B later. + +## Next step: fold into the variation system + +The goal now is to promote this POC into a first-class `Variation` under the system being built per [2026_04_21_variation_system_plan.md](2026_04_21_variation_system_plan.md). Concretely: + +- **New variation class** (e.g. `ObjectColorVariation`) declared as an `available_variation` on `Object` (or a mix-in available to all USD-backed objects). +- **Sampler**: reuse `UniformSampler` for continuous RGB ranges, possibly add a `DiscreteChoiceSampler` for discrete palettes (matches the two code paths `randomize_visual_color` already supports). +- **`build_event_cfg(scene)`**: emit an `EventTermCfg(func=mdp.randomize_visual_color, mode="prestartup", params={...})` with `asset_cfg = SceneEntityCfg(object.name)` and the sampled `colors` spec. +- **Preconditions**: assert `scene.replicate_physics is False` in the variation's setup path, with a clear error message (Newton preset flips it on automatically — see `ArenaEnvBuilder.compose_manager_cfg`). +- **User API sketch**: + ```python + cracker_box.set_variation("color", UniformSampler(low=(0.0,)*3, high=(1.0,)*3)) + ``` + The builder collects these the same way it will collect `ObjectMassVariation` and merges them into `events_cfg`. + +Doing this through the variation system also removes the notebook-specific `env_cfg.events.cracker_box_color = ...` plumbing — users declare the variation on the asset and the builder wires it up. + +## TODOs + +- [ ] **Get `randomize_visual_diffuse_tint` (in-place tint) working** — investigation needed on why the shader-input writes don't visibly affect the render. Candidate causes (in rough priority order): + 1. The YCB assets may have a common `/World/Looks/` material (not per-env), so writing to it doesn't diverge per env. + 2. OmniPBR's `diffuse_tint` may require the texture graph to sample through the tint — asset shaders may route the texture directly to `diffuse_color_constant` instead, making `diffuse_tint` a no-op. Writing `diffuse_color_constant` (while a texture is connected) is a cleaner knob to try next. + 3. The MDL shader input may need to be set on the material prim rather than the shader prim (check `info:mdl:sourceAsset` vs MDL parameter spec). + 4. Instanceability may have been disabled too late — the material resolution might have been cached before `SetInstanceable(False)` ran. + A useful next session is to open the cracker_box USD in Isaac Sim's stage inspector, find the actual shader path, and experiment manually in the Script Editor before re-coding. +- [ ] **Promote the color variation into `ObjectColorVariation`** per the plan above. +- [ ] Decide whether `mdp.randomize_visual_color` (replacement, texture lost) or the in-place tint (texture preserved) is the primary path. Ideally the variation system lets the user choose between them via the variation's constructor args. +- [ ] Add a regression test that spins up a small scene with `num_envs>=2` and asserts the material bindings or shader inputs differ across envs. (Rendering-based assertions would be heavier — a stage-level assertion is probably sufficient.) diff --git a/2026_04_21_variation_system_plan.md b/2026_04_21_variation_system_plan.md new file mode 100644 index 000000000..a8dee1418 --- /dev/null +++ b/2026_04_21_variation_system_plan.md @@ -0,0 +1,142 @@ +# Variation System — Mass POC Plan + +Companion to [2026_04_13_sensitivity_analysis.md](2026_04_13_sensitivity_analysis.md). This plan covers the first slice of the sensitivity-analysis feature: the *variation system* only (analysis tooling is deferred). + +## Goal + +Build the skeleton of a variation system (samplers + variation base + registry) and validate it end-to-end with **one** concrete variation: `ObjectMassVariation` using a `UniformSampler` with hardcoded parameters. No CLI, no eval_runner config, no other variations. + +## Design overview + +Variations are declared by **asset classes** (analogous to how `ObjectBase` already declares event terms via `get_event_cfg()`). An asset class advertises which variations it *supports*; an asset instance holds the set of variations the user has *enabled* on it by supplying a sampler. The builder collects all enabled variations scene-wide and merges their event terms into `events_cfg`. + +```mermaid +flowchart LR + Cls["Object subclass\navailable_variations = {mass: ObjectMassVariation}"] -.declares.-> Inst + User[User code] -->|"obj.set_variation('mass', UniformSampler(0.1, 1.0))"| Inst[Object instance\n_enabled_variations] + Inst --> Scene["Scene.get_variations()"] + EnvExtras[IsaacLabArenaEnvironment.variations\nscene-wide escape hatch] --> Builder + Scene --> Builder[ArenaEnvBuilder.compose_manager_cfg] + Builder -->|"variation.build_event_cfg(scene)"| ETC[EventTermCfg randomize_object_mass] + ETC --> Events[events_cfg] + Events -->|at reset, per env_ids| Fn[randomize_object_mass term fn] + Fn -->|samples| S[UniformSampler.sample num_envs] + Fn -->|writes| Sim[RigidObject.root_physx_view.set_masses] +``` + +Key separations: + +- **Sampler**: "how to draw values" (stateless distribution object). Seeded via an RNG passed in at sample time. +- **Variation**: "what knob to turn + how to turn it". Owns a sampler and an asset target; knows how to emit an `EventTermCfg`. +- **Asset class**: declares *supported* variations (class-level capability). Does **not** enable anything by default. +- **Asset instance**: holds *enabled* variations (user-configured sampler present). Default state = no variation (deterministic). +- **Registry**: global `name → Variation class` table. Not consumed by anything in this POC but establishes the naming contract for later CLI resolution. +- **Integration**: one new step in `ArenaEnvBuilder.compose_manager_cfg` collects `scene.get_variations() + arena_env.variations` and merges their event terms into `events_cfg`. + +## New module layout + +- `isaaclab_arena/variations/__init__.py` — re-exports; triggers registrations via imports. +- `isaaclab_arena/variations/sampler.py` — `Sampler` ABC + `UniformSampler`. +- `isaaclab_arena/variations/base.py` — `Variation` ABC + `VariationRegistry` + `@register_variation` decorator. +- `isaaclab_arena/variations/object_mass.py` — `ObjectMassVariation` + event term function `randomize_object_mass`. + +## Core interfaces (sketch) + +```python +class Sampler(abc.ABC): + @abc.abstractmethod + def sample(self, num_samples: int, generator: torch.Generator) -> torch.Tensor: ... + +class UniformSampler(Sampler): + def __init__(self, low: float, high: float): ... + +class Variation(abc.ABC): + @abc.abstractmethod + def build_event_cfg(self, scene: Scene) -> dict[str, EventTermCfg]: ... + +@register_variation("mass") +class ObjectMassVariation(Variation): + def __init__(self, asset_name: str, sampler: Sampler): ... + def build_event_cfg(self, scene): ... # returns {"_mass_variation": EventTermCfg(...)} +``` + +`randomize_object_mass(env, env_ids, asset_cfg, sampler)`: mirrors Isaac Lab's `randomize_rigid_body_mass` shape but pulls values from our `Sampler` so the variation controls the RNG (see [isaaclab_arena/terms/events.py](isaaclab_arena/terms/events.py) for the existing per-env event pattern, e.g. `set_object_pose_per_env`). Writes masses via `rigid_object.root_physx_view.set_masses(...)`. + +### Asset-declared variation support + +On [isaaclab_arena/assets/object_base.py](isaaclab_arena/assets/object_base.py): + +```python +class ObjectBase: + @classmethod + def available_variations(cls) -> dict[str, type[Variation]]: + return {} # subclasses extend + + def set_variation(self, name: str, sampler: Sampler) -> None: + v_cls = type(self).available_variations()[name] # KeyError if unsupported + self._enabled_variations[name] = v_cls(asset_name=self.name, sampler=sampler) + + def get_variations(self) -> list[Variation]: + return list(self._enabled_variations.values()) +``` + +On [isaaclab_arena/assets/object.py](isaaclab_arena/assets/object.py) (rigid-body `Object`): + +```python +class Object(ObjectBase): + @classmethod + def available_variations(cls): + return {**super().available_variations(), "mass": ObjectMassVariation} +``` + +Every existing rigid object (e.g. `cracker_box`) picks up mass variation support for free. Articulated / non-rigid subclasses can extend the mapping with their own variations later (e.g. `joint_stiffness`). + +`Scene.get_variations()` walks assets and concatenates their enabled variations. + +## Integration touchpoints + +- [isaaclab_arena/assets/object_base.py](isaaclab_arena/assets/object_base.py) — add `available_variations()`, `set_variation()`, `get_variations()`, and the `_enabled_variations` instance dict. +- [isaaclab_arena/assets/object.py](isaaclab_arena/assets/object.py) — extend `available_variations()` with `"mass": ObjectMassVariation` so every rigid object supports it. +- [isaaclab_arena/scene/scene.py](isaaclab_arena/scene/scene.py) — add `get_variations()` walking its assets. +- [isaaclab_arena/environments/isaaclab_arena_environment.py](isaaclab_arena/environments/isaaclab_arena_environment.py) — add an env-level `variations: list[Variation]` escape hatch (and a helper `add_variation`) for scene-wide variations that don't belong to any single asset. +- [isaaclab_arena/environments/arena_env_builder.py](isaaclab_arena/environments/arena_env_builder.py) — in `compose_manager_cfg`, after the existing event merges (`embodiment → scene → task → placement`), collect `scene.get_variations() + arena_env.variations` and merge each one's `build_event_cfg(scene)` into `events_cfg` using the same `combine_configclass_instances` pattern. Conflict policy for this POC: reject duplicate event term names with a clear assert. + +## Example usage (driver script/test, no CLI yet) + +```python +arena_env = KitchenPickAndPlaceEnvironment(args_cli).get_env(args_cli) + +cracker_box = arena_env.scene.get_asset("cracker_box") +cracker_box.set_variation("mass", UniformSampler(0.1, 1.0)) + +env = ArenaEnvBuilder(arena_env, args_cli).make_registered() +``` + +## Tests + +Add [isaaclab_arena/tests/test_variations.py](isaaclab_arena/tests/test_variations.py): + +- **Unit** (no sim): `UniformSampler.sample(n)` returns shape `(n,)`, all values in `[low, high]`, reproducible under a fixed `torch.Generator` seed. +- **Unit** (no sim): `VariationRegistry` register/lookup round-trips; duplicate registration raises. +- **Unit** (no sim): `Object.available_variations()` includes `"mass"`; `set_variation("mass", sampler)` populates `get_variations()` with an `ObjectMassVariation`; `set_variation("nonexistent", ...)` raises. +- **Integration** (sim, mirrors [test_placement_events.py](isaaclab_arena/tests/test_placement_events.py) inner/outer pattern): build a minimal pick-and-place env, call `cracker_box.set_variation("mass", UniformSampler(0.1, 1.0))` *before* building, reset once, read back each env's cracker_box mass via the rigid object view, assert: (a) values lie in `[0.1, 1.0]`, (b) with `num_envs >= 2` and a fixed seed the values are not all identical. + +## Todos + +- [ ] **samplers** — Add `isaaclab_arena/variations/sampler.py` with `Sampler` ABC and `UniformSampler`. +- [ ] **base** — Add `isaaclab_arena/variations/base.py` with `Variation` ABC, `VariationRegistry`, and `@register_variation` decorator. +- [ ] **mass_variation** — Add `isaaclab_arena/variations/object_mass.py` with `ObjectMassVariation` and the `randomize_object_mass` event term function. +- [ ] **asset_support** — Add `available_variations()` / `set_variation()` / `get_variations()` to `ObjectBase` and wire `mass` into the rigid-body `Object` class; have `Scene.get_variations()` collect enabled variations from assets. +- [ ] **env_field** — Add env-level `variations` list + `add_variation` helper to `IsaacLabArenaEnvironment` as an escape hatch for scene-wide variations. +- [ ] **builder_hook** — Integrate variations into `ArenaEnvBuilder.compose_manager_cfg` by merging scene+env variation event terms into `events_cfg`. +- [ ] **package_init** — Wire `isaaclab_arena/variations/__init__.py` to re-export and trigger registrations. +- [ ] **tests** — Add `isaaclab_arena/tests/test_variations.py` with sampler unit tests, registry unit tests, an `Object.set_variation` unit test, and a sim integration test for the mass variation. + +## Out of scope (explicit) + +- CLI syntax (`--variation` flag, dotted-key parsing) — deferred. +- Other samplers (`Choose`, `Normal`) — only stub `Sampler` ABC so they slot in later. +- Other variations (pose, light, camera, HDR, object name) — deferred. +- Semantic target indirection ("pick_up_object" → asset) — use the concrete scene-entity name for now. +- Logging per-env sampled values for downstream sensitivity analysis — deferred. +- Registration-time vs run-time orchestration — this POC only handles per-env runtime variation. diff --git a/isaaclab_arena/examples/compile_env_notebook.py b/isaaclab_arena/examples/compile_env_notebook.py index 109f9f17a..2ffb2569d 100644 --- a/isaaclab_arena/examples/compile_env_notebook.py +++ b/isaaclab_arena/examples/compile_env_notebook.py @@ -11,15 +11,27 @@ import pinocchio # noqa: F401 from isaaclab.app import AppLauncher +from isaaclab_arena.cli.isaaclab_arena_cli import get_isaaclab_arena_cli_parser + print("Launching simulation app once in notebook") -simulation_app = AppLauncher() +# headless=False enables the Kit viewer window so we can visually verify per-env +# randomizations (e.g. object tints). Set headless=True for CI / non-GUI runs. +args_cli = get_isaaclab_arena_cli_parser().parse_args([]) +args_cli.headless = False +args_cli.visualizer = "kit" +app_launcher = AppLauncher(args_cli) +simulation_app = app_launcher.app # %% +import isaaclab.envs.mdp as mdp # noqa: F401 (kept for the commented-out replacement term below) +from isaaclab.managers import EventTermCfg, SceneEntityCfg + from isaaclab_arena.assets.registries import AssetRegistry from isaaclab_arena.cli.isaaclab_arena_cli import get_isaaclab_arena_cli_parser from isaaclab_arena.environments.arena_env_builder import ArenaEnvBuilder from isaaclab_arena.environments.isaaclab_arena_environment import IsaacLabArenaEnvironment +from isaaclab_arena.examples.tint_events import randomize_visual_diffuse_tint from isaaclab_arena.relations.relations import IsAnchor, On from isaaclab_arena.scene.scene import Scene from isaaclab_arena.utils.pose import Pose @@ -31,7 +43,7 @@ cracker_box = asset_registry.get_asset_by_name("cracker_box")() tomato_soup_can = asset_registry.get_asset_by_name("tomato_soup_can")() -cracker_box.set_initial_pose(Pose(position_xyz=(0.4, 0.0, 0.1), rotation_wxyz=(1.0, 0.0, 0.0, 0.0))) +cracker_box.set_initial_pose(Pose(position_xyz=(0.4, 0.0, 0.1), rotation_xyzw=(0.0, 0.0, 0.0, 1.0))) cracker_box.add_relation(IsAnchor()) tomato_soup_can.add_relation(On(cracker_box)) @@ -44,14 +56,82 @@ args_cli = get_isaaclab_arena_cli_parser().parse_args([]) args_cli.solve_relations = True +# Bump num_envs so we can visually verify per-env color variation. +args_cli.num_envs = 4 +args_cli.visualizer = "kit" env_builder = ArenaEnvBuilder(isaaclab_arena_environment, args_cli) -env = env_builder.make_registered() + +# Build the env_cfg so we can inject extra event terms before registration. +env_cfg = env_builder.compose_manager_cfg() + +# %% + +# Per-env visual tint via a custom event (see ``tint_events.py``). +# +# This *keeps* the asset's original diffuse texture and multiplies a random +# color onto it, rather than replacing the material with a flat OmniPBR +# instance like ``mdp.randomize_visual_color`` does. +# +# Requirements: +# * ``scene.replicate_physics`` must be False (Arena default) — with replication +# on, every env shares a single source material and per-env tinting is +# impossible. The event itself also asserts this. +# * ``mode="prestartup"`` → each env gets a stable tint for the entire run. +# Use ``mode="reset"`` instead to resample on every episode reset. +# * The colors dict specifies uniform ranges per channel. Narrow ranges near +# 1.0 (e.g. (0.4, 1.0)) give subtle, photo-realistic tints; wide ranges +# like (0.0, 1.0) look more aggressive. +assert ( + env_cfg.scene.replicate_physics is False +), "randomize_visual_diffuse_tint requires replicate_physics=False; got True." +# env_cfg.events.cracker_box_tint = EventTermCfg( +# func=randomize_visual_diffuse_tint, +# mode="reset", +# params={ +# "asset_cfg": SceneEntityCfg(cracker_box.name), +# "colors": {"r": (0.4, 1.0), "g": (0.4, 1.0), "b": (0.4, 1.0)}, +# }, +# ) +# env_cfg.events.tomato_soup_can_tint = EventTermCfg( +# func=randomize_visual_diffuse_tint, +# mode="prestartup", +# params={ +# "asset_cfg": SceneEntityCfg(tomato_soup_can.name), +# "colors": {"r": (0.4, 1.0), "g": (0.4, 1.0), "b": (0.4, 1.0)}, +# }, +# ) + +# --- Previous behavior: replace the material entirely (texture is lost) --- +env_cfg.events.cracker_box_color = EventTermCfg( + func=mdp.randomize_visual_color, + mode="reset", + params={ + "asset_cfg": SceneEntityCfg(cracker_box.name), + "colors": {"r": (0.0, 1.0), "g": (0.0, 1.0), "b": (0.0, 1.0)}, + "mesh_name": "", + "event_name": "cracker_box_color", + }, +) +# env_cfg.events.tomato_soup_can_color = EventTermCfg( +# func=mdp.randomize_visual_color, +# mode="prestartup", +# params={ +# "asset_cfg": SceneEntityCfg(tomato_soup_can.name), +# "colors": [(1.0, 0.0, 0.0), (0.0, 1.0, 0.0), (0.0, 0.0, 1.0), (1.0, 1.0, 0.0)], +# "mesh_name": "", +# "event_name": "tomato_soup_can_color", +# }, +# ) + +# %% + +env = env_builder.make_registered(env_cfg) env.reset() # %% # Run some zero actions. -NUM_STEPS = 1000 +NUM_STEPS = 500 for _ in tqdm.tqdm(range(NUM_STEPS)): with torch.inference_mode(): actions = torch.zeros(env.action_space.shape, device=env.unwrapped.device) diff --git a/isaaclab_arena/examples/tint_events.py b/isaaclab_arena/examples/tint_events.py new file mode 100644 index 000000000..e4ef452da --- /dev/null +++ b/isaaclab_arena/examples/tint_events.py @@ -0,0 +1,173 @@ +# Copyright (c) 2025-2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md). +# All rights reserved. +# +# SPDX-License-Identifier: Apache-2.0 + +"""Per-env visual tint event that preserves the asset's original texture. + +Unlike :class:`isaaclab.envs.mdp.randomize_visual_color`, which *replaces* the +bound material with a fresh ``OmniPBR`` instance (dropping any diffuse +texture), this term writes a random tint color onto the material that the USD +asset already has bound. For textured assets the effect is a hue shift applied +on top of the original texture. + +The term is scoped to the prototype in ``compile_env_notebook.py`` and kept +alongside it so it's easy to iterate without touching the main arena package. +""" + +from __future__ import annotations + +import re +import torch +from typing import TYPE_CHECKING + +from isaaclab.managers import ManagerTermBase, SceneEntityCfg +from isaaclab.sim.utils import get_current_stage +from pxr import Gf, Sdf, UsdShade + +if TYPE_CHECKING: + from isaaclab.envs import ManagerBasedEnv + from isaaclab.managers.manager_term_cfg import EventTermCfg + + +class randomize_visual_diffuse_tint(ManagerTermBase): + """Randomize the diffuse tint of each env's copy of an asset in place. + + Finds the material bound to each Mesh prim under the asset (per env) and + writes a random color to the bound shader's tint input. The material is + **not** replaced, so any diffuse texture is preserved — the tint simply + multiplies with it. + + Supported shaders: + * ``UsdPreviewSurface``: writes ``inputs:diffuseColor``. + * MDL shaders (e.g. ``OmniPBR``): writes ``inputs:diffuse_tint``. + + Requirements: + * ``env.cfg.scene.replicate_physics`` must be False. With replication on, + every env shares a single source material and per-env tinting is + impossible. + * Run with ``mode="prestartup"`` for a stable tint per env, or + ``mode="reset"`` to resample on every episode reset (note: the current + implementation resamples for *all* envs, matching the ignored-``env_ids`` + behavior of :class:`~isaaclab.envs.mdp.randomize_visual_color`). + + Params (on the :class:`EventTermCfg`): + * ``asset_cfg`` (:class:`SceneEntityCfg`): which scene asset to tint. + * ``colors`` (``dict[str, tuple[float, float]]``): per-channel uniform + ranges, e.g. ``{"r": (0.4, 1.0), "g": (0.4, 1.0), "b": (0.4, 1.0)}``. + Narrow ranges near 1.0 produce subtle tints; wide ranges look + aggressive. + """ + + def __init__(self, cfg: EventTermCfg, env: ManagerBasedEnv): + super().__init__(cfg, env) + + if env.cfg.scene.replicate_physics: + raise RuntimeError( + "randomize_visual_diffuse_tint requires " + "scene.replicate_physics=False (all env clones otherwise share " + "a single source material)." + ) + + asset_cfg: SceneEntityCfg = cfg.params["asset_cfg"] + asset = env.scene[asset_cfg.name] + + # ``asset.cfg.prim_path`` is already a regex-ready string such as + # ``/World/envs/env_.*/cracker_box``. We match descendants by anchoring + # the pattern and requiring a trailing ``/`` so that sibling prims + # with a common prefix (e.g. ``cracker_box_2``) don't match. + prim_path_re = re.compile("^" + asset.cfg.prim_path + "(/|$)") + + stage = get_current_stage() + # One shader-tint target per env. If an asset has multiple textured + # meshes we tint all of them — they'll share the same random color + # per env, which is what a human would expect for a uniform "tint". + self._shader_targets: list[tuple[str, str]] = [] # (shader_path, attr_name) + seen_materials: set[str] = set() + + for prim in stage.Traverse(): + prim_path_str = str(prim.GetPath()) + if not prim_path_re.match(prim_path_str): + continue + if prim.GetTypeName() != "Mesh": + continue + + # Make sure the cloned mesh isn't an instance (otherwise all envs + # share a single material, defeating per-env tinting). + if prim.IsInstanceable(): + prim.SetInstanceable(False) + + mbapi = UsdShade.MaterialBindingAPI(prim) + material = mbapi.ComputeBoundMaterial()[0] + if not material: + continue + + material_path = str(material.GetPath()) + # Avoid writing to the same material twice if multiple meshes + # under the same env share one material. + if material_path in seen_materials: + continue + seen_materials.add(material_path) + + # The surface output of a Material prim drives a Shader prim. + surface_output = material.GetSurfaceOutput() + source = surface_output.GetConnectedSource() if surface_output else None + if not source: + continue + shader_api = source[0] + shader_prim = shader_api.GetPrim() + + attr_name = _tint_attribute_for_shader(shader_prim) + if attr_name is None: + continue + self._shader_targets.append((str(shader_prim.GetPath()), attr_name)) + + self._num_envs = env.scene.num_envs + + def __call__( + self, + env: ManagerBasedEnv, + env_ids: torch.Tensor, + asset_cfg: SceneEntityCfg, + colors: dict[str, tuple[float, float]], + ) -> None: + if not self._shader_targets: + return + + low = torch.tensor([colors["r"][0], colors["g"][0], colors["b"][0]]) + high = torch.tensor([colors["r"][1], colors["g"][1], colors["b"][1]]) + samples = low + (high - low) * torch.rand((len(self._shader_targets), 3)) + + stage = get_current_stage() + for (shader_path, attr_name), rgb in zip(self._shader_targets, samples.tolist()): + shader_prim = stage.GetPrimAtPath(shader_path) + if not shader_prim.IsValid(): + continue + attr = shader_prim.GetAttribute(attr_name) + if not attr: + # Create the attribute lazily; OmniPBR shader inputs aren't + # authored until they're overridden. + attr = shader_prim.CreateAttribute(attr_name, Sdf.ValueTypeNames.Color3f, custom=False) + attr.Set(Gf.Vec3f(*rgb)) + + +def _tint_attribute_for_shader(shader_prim) -> str | None: + """Return the shader input used to tint a surface, or ``None`` if unknown. + + * ``UsdPreviewSurface`` uses ``inputs:diffuseColor`` (which, when a diffuse + texture is connected, is multiplied with the texture sample). + * MDL shaders (OmniPBR and friends) expose ``inputs:diffuse_tint``. + """ + info_id_attr = shader_prim.GetAttribute("info:id") + info_id = info_id_attr.Get() if info_id_attr else None + if info_id == "UsdPreviewSurface": + return "inputs:diffuseColor" + + # MDL shaders advertise their source through ``info:mdl:sourceAsset``. + mdl_source_attr = shader_prim.GetAttribute("info:mdl:sourceAsset") + if mdl_source_attr and mdl_source_attr.Get(): + return "inputs:diffuse_tint" + + # Unknown shader type — skip rather than scribble attributes that won't + # be consumed by any rendering path. + return None From 9e5c71c0f9206381bb0a809ae0a5bf7bdbfe75cc Mon Sep 17 00:00:00 2001 From: alex Date: Wed, 22 Apr 2026 15:26:24 +0200 Subject: [PATCH 02/25] Prototype of the variations interface. --- 2026_04_21_variations_user_interface.py | 22 ++++ isaaclab_arena/variations/__init__.py | 24 ++++ isaaclab_arena/variations/object_color.py | 117 ++++++++++++++++++ isaaclab_arena/variations/sampler.py | 89 +++++++++++++ isaaclab_arena/variations/variation_base.py | 53 ++++++++ .../variations/variation_registry.py | 70 +++++++++++ 6 files changed, 375 insertions(+) create mode 100644 2026_04_21_variations_user_interface.py create mode 100644 isaaclab_arena/variations/__init__.py create mode 100644 isaaclab_arena/variations/object_color.py create mode 100644 isaaclab_arena/variations/sampler.py create mode 100644 isaaclab_arena/variations/variation_base.py create mode 100644 isaaclab_arena/variations/variation_registry.py diff --git a/2026_04_21_variations_user_interface.py b/2026_04_21_variations_user_interface.py new file mode 100644 index 000000000..0b007ba48 --- /dev/null +++ b/2026_04_21_variations_user_interface.py @@ -0,0 +1,22 @@ +# Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md). +# All rights reserved. +# +# SPDX-License-Identifier: Apache-2.0 + +## + + +# Option 1: Variations travel with asset. +# Decision: AGAINST + +asset_registry = AssetRegistry() +apple = asset_registry.get_asset_by_name("apple") +apple.get_variation("color").enable() +apple.get_variation("color").set_sampler(UniformSampler(low=(0.0,) * 3, high=(1.0,) * 3)) + +# Option 2: Variations are added objects. +# Decision: SUPPORTED + +asset_registry = AssetRegistry() +apple = asset_registry.get_asset_by_name("apple") +color_variation = ObjectColorVariation(apple, sampler=UniformSampler(low=(0.0,) * 3, high=(1.0,) * 3)) diff --git a/isaaclab_arena/variations/__init__.py b/isaaclab_arena/variations/__init__.py new file mode 100644 index 000000000..00ab99aa7 --- /dev/null +++ b/isaaclab_arena/variations/__init__.py @@ -0,0 +1,24 @@ +# Copyright (c) 2025-2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md). +# All rights reserved. +# +# SPDX-License-Identifier: Apache-2.0 + +"""Variation system public API. + +Importing this package triggers registration of all built-in variations in +:class:`~isaaclab_arena.variations.base.VariationRegistry`. +""" + +from isaaclab_arena.variations.object_color import ObjectColorVariation +from isaaclab_arena.variations.sampler import Sampler, UniformSampler +from isaaclab_arena.variations.variation_base import VariationBase +from isaaclab_arena.variations.variation_registry import VariationRegistry, register_variation + +__all__ = [ + "ObjectColorVariation", + "Sampler", + "UniformSampler", + "VariationBase", + "VariationRegistry", + "register_variation", +] diff --git a/isaaclab_arena/variations/object_color.py b/isaaclab_arena/variations/object_color.py new file mode 100644 index 000000000..0356b241d --- /dev/null +++ b/isaaclab_arena/variations/object_color.py @@ -0,0 +1,117 @@ +# Copyright (c) 2025-2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md). +# All rights reserved. +# +# SPDX-License-Identifier: Apache-2.0 + +"""Per-asset visual color variation. + +Wraps :class:`isaaclab.envs.mdp.randomize_visual_color` — the "replace the +bound material" path validated in ``isaaclab_arena/examples/compile_env_notebook.py``. +Each cloned env ends up with a distinct random flat color; the asset's +original diffuse texture is dropped (the in-place tint path remains a TODO, +see ``2026_04_21_color_variation_status.md``). + +Sampler support in this POC is limited to a 3D :class:`UniformSampler` over +RGB. :class:`~isaaclab.envs.mdp.randomize_visual_color` samples internally +from the ``colors`` dict we build here; the sampler's bounds are forwarded +as-is. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +import isaaclab.envs.mdp as mdp +from isaaclab.managers import EventTermCfg, SceneEntityCfg + +from isaaclab_arena.variations.sampler import Sampler, UniformSampler +from isaaclab_arena.variations.variation_base import VariationBase +from isaaclab_arena.variations.variation_registry import register_variation + +if TYPE_CHECKING: + from isaaclab_arena.assets.object_base import ObjectBase + from isaaclab_arena.scene.scene import Scene + + +@register_variation("color") +class ObjectColorVariation(VariationBase): + """Randomize an object's visual color per env. + + Emits a single :class:`EventTermCfg` bound to + :class:`isaaclab.envs.mdp.randomize_visual_color`. The target asset's + bound material is replaced with a fresh ``OmniPBR`` instance whose + ``diffuse_color_constant`` is sampled (uniformly over RGB) from the + bounds of the provided :class:`UniformSampler`. + + Requirements: + * ``scene.replicate_physics`` must be False (the Arena default). + With replication on, all envs share one material and per-env + randomization is impossible. :class:`randomize_visual_color` + asserts this at construction time. + + Args: + asset: The :class:`~isaaclab_arena.assets.object_base.ObjectBase` + (or subclass) instance whose visual color will be varied. The + asset's ``name`` is used to resolve the scene entity at event + time, so the same instance must also be registered on the + :class:`~isaaclab_arena.scene.scene.Scene`. + sampler: Distribution over RGB triples. Currently restricted to + a :class:`UniformSampler` with ``event_shape == (3,)``. + mode: Event mode. ``"reset"`` resamples on every episode reset; + ``"prestartup"`` picks a stable color per env for the whole run. + mesh_name: Sub-mesh selector forwarded to + :class:`randomize_visual_color`. Empty string targets all + meshes under the asset's prim. + """ + + def __init__( + self, + asset: ObjectBase, + sampler: Sampler, + mode: str = "reset", + mesh_name: str = "", + ): + self.asset = asset + self.sampler = sampler + self.mode = mode + self.mesh_name = mesh_name + + def build_event_cfg(self, scene: Scene) -> tuple[str, EventTermCfg]: # noqa: ARG002 + colors = self._sampler_to_colors_spec() + event_name = f"{self.asset.name}_color_variation" + event_cfg = EventTermCfg( + func=mdp.randomize_visual_color, + mode=self.mode, + params={ + "asset_cfg": SceneEntityCfg(self.asset.name), + "colors": colors, + "mesh_name": self.mesh_name, + "event_name": event_name, + }, + ) + return event_name, event_cfg + + def _sampler_to_colors_spec(self) -> dict[str, tuple[float, float]]: + """Translate ``self.sampler`` into the ``colors`` dict the event term expects. + + :class:`randomize_visual_color` accepts either a list of discrete + RGB triples or a dict ``{"r": (low, high), "g": (...), "b": (...)}`` + of per-channel uniform ranges. We currently only produce the dict + form from a 3D :class:`UniformSampler`; richer sampler types (e.g. + a discrete choice sampler) can extend this mapping later. + """ + assert isinstance(self.sampler, UniformSampler), ( + f"ObjectColorVariation currently only supports UniformSampler; got {type(self.sampler).__name__}. " + "Discrete palette support (DiscreteChoiceSampler) is planned but not implemented." + ) + assert tuple(self.sampler.event_shape) == (3,), ( + "ObjectColorVariation expects a 3D UniformSampler over RGB; got event_shape " + f"{tuple(self.sampler.event_shape)}." + ) + low = self.sampler.low.tolist() + high = self.sampler.high.tolist() + return { + "r": (low[0], high[0]), + "g": (low[1], high[1]), + "b": (low[2], high[2]), + } diff --git a/isaaclab_arena/variations/sampler.py b/isaaclab_arena/variations/sampler.py new file mode 100644 index 000000000..5f1cbb657 --- /dev/null +++ b/isaaclab_arena/variations/sampler.py @@ -0,0 +1,89 @@ +# Copyright (c) 2025-2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md). +# All rights reserved. +# +# SPDX-License-Identifier: Apache-2.0 + +"""Samplers for the variation system. + +A :class:`Sampler` is a stateless description of *how* values are drawn. It +does not own any RNG — instead, callers pass a :class:`torch.Generator` at +sample time so the variation system can control seeding centrally. + +Concrete samplers live in this module so they slot in uniformly: a variation +receives a sampler, inspects it if it needs to translate it for a foreign +API (see e.g. :class:`~isaaclab_arena.variations.object_color.ObjectColorVariation`), +or just calls :meth:`Sampler.sample` on it at event time. +""" + +from __future__ import annotations + +import torch +from abc import ABC, abstractmethod +from collections.abc import Sequence + + +class Sampler(ABC): + """Abstract distribution over values. + + Implementations are expected to be stateless: all randomness flows + through the ``generator`` argument, so repeated sampling with the same + generator state is reproducible. + """ + + @abstractmethod + def sample(self, num_samples: int, generator: torch.Generator | None = None) -> torch.Tensor: + """Draw ``num_samples`` values from this distribution. + + Args: + num_samples: Number of independent samples to draw. Must be ``>= 0``. + generator: Optional generator to pull randomness from. If ``None``, + the default torch RNG is used. + + Returns: + Tensor of shape ``(num_samples, *event_shape)`` where + ``event_shape`` is empty for scalar samplers and ``(d,)`` for + vector-valued samplers (e.g. an RGB uniform). + """ + ... + + +class UniformSampler(Sampler): + """Uniform sampler over ``[low, high]``. + + ``low`` and ``high`` may be scalars (producing scalar samples) or + broadcast-compatible sequences (producing vector samples of the same + shape). The bounds are inclusive; samples are drawn with + ``low + (high - low) * U(0, 1)``. + + Examples: + Scalar mass range:: + + UniformSampler(0.1, 1.0) + + Per-channel RGB range:: + + UniformSampler(low=(0.0, 0.0, 0.0), high=(1.0, 1.0, 1.0)) + """ + + def __init__(self, low: float | Sequence[float], high: float | Sequence[float]): + low_t = torch.as_tensor(low, dtype=torch.float32) + high_t = torch.as_tensor(high, dtype=torch.float32) + assert ( + low_t.shape == high_t.shape + ), f"UniformSampler low/high must have matching shape; got {tuple(low_t.shape)} vs {tuple(high_t.shape)}." + assert torch.all( + low_t <= high_t + ), f"UniformSampler requires low <= high elementwise; got low={low_t}, high={high_t}." + self.low = low_t + self.high = high_t + + @property + def event_shape(self) -> torch.Size: + """Shape of a single sample (empty for scalar samplers).""" + return self.low.shape + + def sample(self, num_samples: int, generator: torch.Generator | None = None) -> torch.Tensor: + assert num_samples >= 0, f"num_samples must be non-negative; got {num_samples}." + shape = (num_samples, *self.event_shape) + u = torch.rand(shape, generator=generator) + return self.low + (self.high - self.low) * u diff --git a/isaaclab_arena/variations/variation_base.py b/isaaclab_arena/variations/variation_base.py new file mode 100644 index 000000000..87ef13774 --- /dev/null +++ b/isaaclab_arena/variations/variation_base.py @@ -0,0 +1,53 @@ +# Copyright (c) 2025-2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md). +# All rights reserved. +# +# SPDX-License-Identifier: Apache-2.0 + +"""Variation abstract base class. + +A :class:`VariationBase` describes *one* knob to turn on the scene — the +target asset (or the scene itself), the sampler that drives it, and the +event term that realises it. The builder collects all enabled variations +and merges their event terms into ``events_cfg``. +""" + +from __future__ import annotations + +from abc import ABC, abstractmethod +from typing import TYPE_CHECKING + +from isaaclab.managers import EventTermCfg + +if TYPE_CHECKING: + from isaaclab_arena.scene.scene import Scene + + +class VariationBase(ABC): + """Abstract variation. + + A variation binds a target (typically an asset name) and a + :class:`~isaaclab_arena.variations.sampler.Sampler` to the event term + required to apply it on reset / prestartup. Subclasses implement + :meth:`build_event_cfg`, which the builder calls once per enabled + variation and whose outputs are merged into the environment's + ``events_cfg``. + """ + + @abstractmethod + def build_event_cfg(self, scene: Scene) -> tuple[str, EventTermCfg]: + """Return the event term that realises this variation. + + Args: + scene: The arena scene; passed in case a variation needs to + inspect or resolve other assets (e.g. a scene-wide light + variation). Simple per-asset variations may ignore it. + + Returns: + ``(name, cfg)`` pair. ``name`` must be unique across *all* + enabled variations in the scene — the builder will raise if it + collides with another event term. Variations that need to fan + out to multiple assets should be expressed as multiple + :class:`VariationBase` instances rather than a single variation + emitting multiple terms. + """ + ... diff --git a/isaaclab_arena/variations/variation_registry.py b/isaaclab_arena/variations/variation_registry.py new file mode 100644 index 000000000..176babed0 --- /dev/null +++ b/isaaclab_arena/variations/variation_registry.py @@ -0,0 +1,70 @@ +# Copyright (c) 2025-2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md). +# All rights reserved. +# +# SPDX-License-Identifier: Apache-2.0 + +"""Global registry of :class:`VariationBase` subclasses. + +Populated at import time via the :func:`register_variation` decorator. +Not consumed by the builder yet — this establishes the naming contract +used by future CLI resolution (e.g. ``--variation cracker_box.color=...``). +""" + +from __future__ import annotations + +from typing import ClassVar + +from isaaclab_arena.variations.variation_base import VariationBase + + +class VariationRegistry: + """Global name → :class:`VariationBase` subclass table. + + Populated at import time via :func:`register_variation`. Duplicate + registrations raise to catch accidental re-registration. + """ + + _entries: ClassVar[dict[str, type[VariationBase]]] = {} + + @classmethod + def register(cls, name: str, variation_cls: type[VariationBase]) -> None: + """Register ``variation_cls`` under ``name``. + + Raises: + ValueError: If ``name`` is already registered. + """ + if name in cls._entries: + raise ValueError( + f"Variation '{name}' is already registered to " + f"{cls._entries[name].__module__}.{cls._entries[name].__name__}." + ) + cls._entries[name] = variation_cls + + @classmethod + def get(cls, name: str) -> type[VariationBase]: + """Return the :class:`VariationBase` subclass registered under ``name``.""" + if name not in cls._entries: + raise KeyError(f"Variation '{name}' is not registered. Known variations: {sorted(cls._entries)}") + return cls._entries[name] + + @classmethod + def entries(cls) -> dict[str, type[VariationBase]]: + """Return a shallow copy of the current registry.""" + return dict(cls._entries) + + +def register_variation(name: str): + """Decorator: register a :class:`VariationBase` subclass under ``name``. + + Example:: + + @register_variation("color") + class ObjectColorVariation(VariationBase): + ... + """ + + def decorator(cls: type[VariationBase]) -> type[VariationBase]: + VariationRegistry.register(name, cls) + return cls + + return decorator From 37ec8daeba2cb687ff5425ccdc8635c2d8d0a353 Mon Sep 17 00:00:00 2001 From: alex Date: Wed, 22 Apr 2026 15:39:55 +0200 Subject: [PATCH 03/25] Add variations to objects. --- 2026_04_21_variations_user_interface.py | 9 +++- isaaclab_arena/assets/object.py | 6 +++ isaaclab_arena/assets/object_base.py | 30 +++++++++++++ isaaclab_arena/variations/object_color.py | 29 ++++++------ isaaclab_arena/variations/variation_base.py | 49 +++++++++++++++++---- 5 files changed, 99 insertions(+), 24 deletions(-) diff --git a/2026_04_21_variations_user_interface.py b/2026_04_21_variations_user_interface.py index 0b007ba48..676613b71 100644 --- a/2026_04_21_variations_user_interface.py +++ b/2026_04_21_variations_user_interface.py @@ -7,7 +7,12 @@ # Option 1: Variations travel with asset. -# Decision: AGAINST +# DECISION: SUPPORTED +# Reason: Certian variations will be specific to a single asset, for example +# embodiment specific variations. Therefore it makes sense that variations +# travel with the asset. The second thing is that then, for default variations +# the user doesn't need to deal with them in the environment file, they are +# automatically there (disabled by default). asset_registry = AssetRegistry() apple = asset_registry.get_asset_by_name("apple") @@ -15,7 +20,7 @@ apple.get_variation("color").set_sampler(UniformSampler(low=(0.0,) * 3, high=(1.0,) * 3)) # Option 2: Variations are added objects. -# Decision: SUPPORTED +# DECISION: AGAINST asset_registry = AssetRegistry() apple = asset_registry.get_asset_by_name("apple") diff --git a/isaaclab_arena/assets/object.py b/isaaclab_arena/assets/object.py index bbc2036b4..80d4b2289 100644 --- a/isaaclab_arena/assets/object.py +++ b/isaaclab_arena/assets/object.py @@ -17,6 +17,8 @@ from isaaclab_arena.utils.pose import Pose from isaaclab_arena.utils.usd.rigid_bodies import find_shallowest_rigid_body from isaaclab_arena.utils.usd_helpers import compute_local_bounding_box_from_usd, has_light, open_stage +from isaaclab_arena.variations.object_color import ObjectColorVariation +from isaaclab_arena.variations.variation_base import VariationBase class Object(ObjectBase): @@ -62,6 +64,10 @@ def __init__( self.object_cfg = self._init_object_cfg() self.event_cfg = self._init_event_cfg() + @classmethod + def available_variations(cls) -> dict[str, type[VariationBase]]: + return {**super().available_variations(), "color": ObjectColorVariation} + def add_relation(self, relation: RelationBase) -> None: """Add a relation to this object.""" self.relations.append(relation) diff --git a/isaaclab_arena/assets/object_base.py b/isaaclab_arena/assets/object_base.py index 2ea9e3fdf..7cec4ea13 100644 --- a/isaaclab_arena/assets/object_base.py +++ b/isaaclab_arena/assets/object_base.py @@ -22,6 +22,7 @@ from isaaclab_arena.utils.bounding_box import AxisAlignedBoundingBox from isaaclab_arena.utils.pose import Pose, PosePerEnv, PoseRange from isaaclab_arena.utils.velocity import Velocity +from isaaclab_arena.variations.variation_base import VariationBase class ObjectType(Enum): @@ -50,6 +51,35 @@ def __init__( self.object_cfg = None self.event_cfg = None self.relations: list[RelationBase] = [] + self._variations: dict[str, VariationBase] = { + variation_name: variation_cls(self) + for variation_name, variation_cls in type(self).available_variations().items() + } + + @classmethod + def available_variations(cls) -> dict[str, type[VariationBase]]: + """Variation classes this asset class supports. + + Subclasses extend via ``{**super().available_variations(), "name": VariationCls}``. + Each entry is instantiated once per asset instance at construction time and + starts disabled; users opt in via + :meth:`get_variation` -> :meth:`~isaaclab_arena.variations.variation_base.VariationBase.enable` + and configure it via :meth:`~isaaclab_arena.variations.variation_base.VariationBase.set_sampler`. + """ + return {} + + def get_variation(self, name: str) -> VariationBase: + """Return the variation with the given name; raises ``KeyError`` if unsupported.""" + if name not in self._variations: + raise KeyError( + f"Asset '{self.name}' ({type(self).__name__}) does not support variation '{name}'. " + f"Supported variations: {sorted(self._variations)}." + ) + return self._variations[name] + + def get_variations(self) -> list[VariationBase]: + """Return all enabled variations declared on this asset.""" + return [v for v in self._variations.values() if v.enabled] def get_initial_pose(self) -> Pose | PoseRange | PosePerEnv | None: """Return the current initial pose of this object. diff --git a/isaaclab_arena/variations/object_color.py b/isaaclab_arena/variations/object_color.py index 0356b241d..1cc27ce5e 100644 --- a/isaaclab_arena/variations/object_color.py +++ b/isaaclab_arena/variations/object_color.py @@ -24,7 +24,7 @@ import isaaclab.envs.mdp as mdp from isaaclab.managers import EventTermCfg, SceneEntityCfg -from isaaclab_arena.variations.sampler import Sampler, UniformSampler +from isaaclab_arena.variations.sampler import UniformSampler from isaaclab_arena.variations.variation_base import VariationBase from isaaclab_arena.variations.variation_registry import register_variation @@ -41,7 +41,8 @@ class ObjectColorVariation(VariationBase): :class:`isaaclab.envs.mdp.randomize_visual_color`. The target asset's bound material is replaced with a fresh ``OmniPBR`` instance whose ``diffuse_color_constant`` is sampled (uniformly over RGB) from the - bounds of the provided :class:`UniformSampler`. + bounds of the sampler provided via + :meth:`~isaaclab_arena.variations.variation_base.VariationBase.set_sampler`. Requirements: * ``scene.replicate_physics`` must be False (the Arena default). @@ -55,8 +56,6 @@ class ObjectColorVariation(VariationBase): asset's ``name`` is used to resolve the scene entity at event time, so the same instance must also be registered on the :class:`~isaaclab_arena.scene.scene.Scene`. - sampler: Distribution over RGB triples. Currently restricted to - a :class:`UniformSampler` with ``event_shape == (3,)``. mode: Event mode. ``"reset"`` resamples on every episode reset; ``"prestartup"`` picks a stable color per env for the whole run. mesh_name: Sub-mesh selector forwarded to @@ -67,16 +66,18 @@ class ObjectColorVariation(VariationBase): def __init__( self, asset: ObjectBase, - sampler: Sampler, mode: str = "reset", mesh_name: str = "", ): - self.asset = asset - self.sampler = sampler + super().__init__(asset) self.mode = mode self.mesh_name = mesh_name def build_event_cfg(self, scene: Scene) -> tuple[str, EventTermCfg]: # noqa: ARG002 + assert self._sampler is not None, ( + f"ObjectColorVariation on '{self.asset.name}' is enabled but no sampler is set; " + "call .set_sampler(...) before building the env." + ) colors = self._sampler_to_colors_spec() event_name = f"{self.asset.name}_color_variation" event_cfg = EventTermCfg( @@ -92,7 +93,7 @@ def build_event_cfg(self, scene: Scene) -> tuple[str, EventTermCfg]: # noqa: AR return event_name, event_cfg def _sampler_to_colors_spec(self) -> dict[str, tuple[float, float]]: - """Translate ``self.sampler`` into the ``colors`` dict the event term expects. + """Translate ``self._sampler`` into the ``colors`` dict the event term expects. :class:`randomize_visual_color` accepts either a list of discrete RGB triples or a dict ``{"r": (low, high), "g": (...), "b": (...)}`` @@ -100,16 +101,16 @@ def _sampler_to_colors_spec(self) -> dict[str, tuple[float, float]]: form from a 3D :class:`UniformSampler`; richer sampler types (e.g. a discrete choice sampler) can extend this mapping later. """ - assert isinstance(self.sampler, UniformSampler), ( - f"ObjectColorVariation currently only supports UniformSampler; got {type(self.sampler).__name__}. " + assert isinstance(self._sampler, UniformSampler), ( + f"ObjectColorVariation currently only supports UniformSampler; got {type(self._sampler).__name__}. " "Discrete palette support (DiscreteChoiceSampler) is planned but not implemented." ) - assert tuple(self.sampler.event_shape) == (3,), ( + assert tuple(self._sampler.event_shape) == (3,), ( "ObjectColorVariation expects a 3D UniformSampler over RGB; got event_shape " - f"{tuple(self.sampler.event_shape)}." + f"{tuple(self._sampler.event_shape)}." ) - low = self.sampler.low.tolist() - high = self.sampler.high.tolist() + low = self._sampler.low.tolist() + high = self._sampler.high.tolist() return { "r": (low[0], high[0]), "g": (low[1], high[1]), diff --git a/isaaclab_arena/variations/variation_base.py b/isaaclab_arena/variations/variation_base.py index 87ef13774..11b983dcb 100644 --- a/isaaclab_arena/variations/variation_base.py +++ b/isaaclab_arena/variations/variation_base.py @@ -6,9 +6,12 @@ """Variation abstract base class. A :class:`VariationBase` describes *one* knob to turn on the scene — the -target asset (or the scene itself), the sampler that drives it, and the -event term that realises it. The builder collects all enabled variations -and merges their event terms into ``events_cfg``. +target asset, the sampler that drives it, and the event term that realises +it. Variations are instantiated by their target asset as part of +:meth:`~isaaclab_arena.assets.object_base.ObjectBase.available_variations` +(disabled + sampler-less by default) and then configured by the user via +:meth:`enable` / :meth:`set_sampler`. The builder walks the scene, collects +enabled variations, and merges their event terms into ``events_cfg``. """ from __future__ import annotations @@ -19,20 +22,50 @@ from isaaclab.managers import EventTermCfg if TYPE_CHECKING: + from isaaclab_arena.assets.object_base import ObjectBase from isaaclab_arena.scene.scene import Scene + from isaaclab_arena.variations.sampler import Sampler class VariationBase(ABC): """Abstract variation. - A variation binds a target (typically an asset name) and a + A variation binds a target asset and a :class:`~isaaclab_arena.variations.sampler.Sampler` to the event term - required to apply it on reset / prestartup. Subclasses implement - :meth:`build_event_cfg`, which the builder calls once per enabled - variation and whose outputs are merged into the environment's - ``events_cfg``. + required to apply it on reset / prestartup. It starts disabled with no + sampler; the user flips it on via :meth:`enable` and supplies a sampler + via :meth:`set_sampler` before the environment is built. Subclasses + implement :meth:`build_event_cfg`, which the builder calls once per + enabled variation. """ + def __init__(self, asset: ObjectBase): + self.asset = asset + self._enabled: bool = False + self._sampler: Sampler | None = None + + @property + def enabled(self) -> bool: + """Whether this variation is active and should be built into ``events_cfg``.""" + return self._enabled + + def enable(self) -> None: + """Mark this variation as active. A sampler must still be provided via :meth:`set_sampler`.""" + self._enabled = True + + def disable(self) -> None: + """Mark this variation as inactive. It will be skipped by the builder.""" + self._enabled = False + + @property + def sampler(self) -> Sampler | None: + """The sampler driving this variation, or ``None`` if not yet set.""" + return self._sampler + + def set_sampler(self, sampler: Sampler) -> None: + """Set the sampler driving this variation.""" + self._sampler = sampler + @abstractmethod def build_event_cfg(self, scene: Scene) -> tuple[str, EventTermCfg]: """Return the event term that realises this variation. From 3d27adb63adba03063447f9324aaa10edfbd161a Mon Sep 17 00:00:00 2001 From: alex Date: Wed, 22 Apr 2026 16:46:58 +0200 Subject: [PATCH 04/25] Variations as concrete classes. --- isaaclab_arena/assets/object.py | 6 +- isaaclab_arena/assets/object_base.py | 32 ++++---- .../environments/arena_env_builder.py | 32 ++++++++ .../examples/compile_env_notebook.py | 78 +++++++------------ isaaclab_arena/scene/scene.py | 11 ++- isaaclab_arena/variations/object_color.py | 16 +++- isaaclab_arena/variations/variation_base.py | 35 ++++++--- .../variations/variation_registry.py | 25 +++--- 8 files changed, 143 insertions(+), 92 deletions(-) diff --git a/isaaclab_arena/assets/object.py b/isaaclab_arena/assets/object.py index 80d4b2289..b584a8ba9 100644 --- a/isaaclab_arena/assets/object.py +++ b/isaaclab_arena/assets/object.py @@ -18,7 +18,6 @@ from isaaclab_arena.utils.usd.rigid_bodies import find_shallowest_rigid_body from isaaclab_arena.utils.usd_helpers import compute_local_bounding_box_from_usd, has_light, open_stage from isaaclab_arena.variations.object_color import ObjectColorVariation -from isaaclab_arena.variations.variation_base import VariationBase class Object(ObjectBase): @@ -63,10 +62,7 @@ def __init__( self.bounding_box = None self.object_cfg = self._init_object_cfg() self.event_cfg = self._init_event_cfg() - - @classmethod - def available_variations(cls) -> dict[str, type[VariationBase]]: - return {**super().available_variations(), "color": ObjectColorVariation} + self.add_variation(ObjectColorVariation(self)) def add_relation(self, relation: RelationBase) -> None: """Add a relation to this object.""" diff --git a/isaaclab_arena/assets/object_base.py b/isaaclab_arena/assets/object_base.py index 7cec4ea13..0978fc71a 100644 --- a/isaaclab_arena/assets/object_base.py +++ b/isaaclab_arena/assets/object_base.py @@ -51,22 +51,24 @@ def __init__( self.object_cfg = None self.event_cfg = None self.relations: list[RelationBase] = [] - self._variations: dict[str, VariationBase] = { - variation_name: variation_cls(self) - for variation_name, variation_cls in type(self).available_variations().items() - } - - @classmethod - def available_variations(cls) -> dict[str, type[VariationBase]]: - """Variation classes this asset class supports. - - Subclasses extend via ``{**super().available_variations(), "name": VariationCls}``. - Each entry is instantiated once per asset instance at construction time and - starts disabled; users opt in via - :meth:`get_variation` -> :meth:`~isaaclab_arena.variations.variation_base.VariationBase.enable` - and configure it via :meth:`~isaaclab_arena.variations.variation_base.VariationBase.set_sampler`. + self._variations: dict[str, VariationBase] = {} + + def add_variation(self, variation: VariationBase) -> None: + """Attach a concrete variation to this asset under its :attr:`VariationBase.name`. + + Subclasses call this from their ``__init__`` (after ``super().__init__``) + to declare the variations they support, e.g. + ``self.add_variation(ObjectColorVariation(self))``. The variation's + class-level ``name`` is used as the key, so it matches the name the + variation is registered under globally (via + :func:`~isaaclab_arena.variations.variation_registry.register_variation`) + and the name users reference through :meth:`get_variation`. Each variation + is added in its default state (disabled, pre-configured with a sensible + default sampler where applicable) so that users can opt in with a single + :meth:`get_variation(name).enable() ` + call. Re-registering the same name overwrites the existing entry. """ - return {} + self._variations[variation.name] = variation def get_variation(self, name: str) -> VariationBase: """Return the variation with the given name; raises ``KeyError`` if unsupported.""" diff --git a/isaaclab_arena/environments/arena_env_builder.py b/isaaclab_arena/environments/arena_env_builder.py index 3b1e6b5d3..dc5d44e7e 100644 --- a/isaaclab_arena/environments/arena_env_builder.py +++ b/isaaclab_arena/environments/arena_env_builder.py @@ -173,6 +173,36 @@ def _apply_pool_layouts_to_objects( else: obj.set_initial_pose(PosePerEnv(poses=poses)) + def _compose_variations_event_cfg(self): + """Build a configclass instance holding an :class:`EventTermCfg` per enabled variation. + + Walks every enabled variation on the scene (and, later, any env-level + variation escape hatch) and asks it for its event term via + :meth:`~isaaclab_arena.variations.variation_base.VariationBase.build_event_cfg`. + Returns ``None`` when nothing is enabled so + :func:`combine_configclass_instances` skips it cleanly. + + Raises: + AssertionError: If two variations want the same event-term name + (variations are responsible for uniquely namespacing their + terms, typically by prefixing with the asset name). + """ + variations = self.arena_env.scene.get_variations() + if not variations: + return None + fields: list[tuple[str, type, EventTermCfg]] = [] + seen: set[str] = set() + for variation in variations: + event_name, event_cfg = variation.build_event_cfg(self.arena_env.scene) + assert event_name not in seen, ( + f"Duplicate variation event term name '{event_name}'. " + "Each variation must produce a unique name; consider prefixing with the asset name." + ) + seen.add(event_name) + fields.append((event_name, EventTermCfg, event_cfg)) + VariationsEventCfg = make_configclass("VariationsEventCfg", fields) + return VariationsEventCfg() + def _modify_recorder_cfg_dataset_filename(self, recorder_cfg: RecorderManagerBaseCfg) -> RecorderManagerBaseCfg: """Modify the recorder dataset filename to include the timestamp and rank.""" base = getattr(recorder_cfg, "dataset_filename", "dataset") @@ -224,12 +254,14 @@ def compose_manager_cfg(self) -> IsaacLabArenaManagerBasedRLEnvCfg: [("placement_reset", EventTermCfg, self._placement_event_cfg)], ) placement_event_cfg = PlacementEventCfg() + variations_event_cfg = self._compose_variations_event_cfg() events_cfg = combine_configclass_instances( "EventsCfg", embodiment.get_events_cfg(), self.arena_env.scene.get_events_cfg(), task.get_events_cfg(), placement_event_cfg, + variations_event_cfg, ) termination_cfg = combine_configclass_instances( "TerminationCfg", diff --git a/isaaclab_arena/examples/compile_env_notebook.py b/isaaclab_arena/examples/compile_env_notebook.py index 2ffb2569d..89ee7f414 100644 --- a/isaaclab_arena/examples/compile_env_notebook.py +++ b/isaaclab_arena/examples/compile_env_notebook.py @@ -24,17 +24,18 @@ # %% -import isaaclab.envs.mdp as mdp # noqa: F401 (kept for the commented-out replacement term below) -from isaaclab.managers import EventTermCfg, SceneEntityCfg +import isaaclab.envs.mdp as mdp # noqa: F401 (kept for the commented-out in-place tint notes below) +from isaaclab.managers import EventTermCfg, SceneEntityCfg # noqa: F401 (kept for the commented-out in-place tint notes below) from isaaclab_arena.assets.registries import AssetRegistry from isaaclab_arena.cli.isaaclab_arena_cli import get_isaaclab_arena_cli_parser from isaaclab_arena.environments.arena_env_builder import ArenaEnvBuilder from isaaclab_arena.environments.isaaclab_arena_environment import IsaacLabArenaEnvironment -from isaaclab_arena.examples.tint_events import randomize_visual_diffuse_tint +from isaaclab_arena.examples.tint_events import randomize_visual_diffuse_tint # noqa: F401 (kept for the commented-out in-place tint notes below) from isaaclab_arena.relations.relations import IsAnchor, On from isaaclab_arena.scene.scene import Scene from isaaclab_arena.utils.pose import Pose +from isaaclab_arena.variations import UniformSampler asset_registry = AssetRegistry() @@ -47,6 +48,19 @@ cracker_box.add_relation(IsAnchor()) tomato_soup_can.add_relation(On(cracker_box)) +# --- New-style variation configuration --------------------------------------- +# +# Every ``Object`` ships with a registry of built-in variations (currently just +# ``"color"``), pre-configured with a sensible default sampler. Calling +# :meth:`~isaaclab_arena.variations.variation_base.VariationBase.enable` alone +# is enough to get reasonable behaviour; :meth:`set_sampler` is only needed to +# narrow or replace the default distribution. +cracker_box.get_variation("color").enable() # uses the default full-RGB sampler + +# Uncomment to also randomize the soup can with a tighter (pastel) range: +# tomato_soup_can.get_variation("color").enable() +# tomato_soup_can.get_variation("color").set_sampler(UniformSampler(low=(0.4,) * 3, high=(1.0,) * 3)) + scene = Scene(assets=[background, cracker_box, tomato_soup_can]) isaaclab_arena_environment = IsaacLabArenaEnvironment( name="reference_object_test", @@ -61,29 +75,21 @@ args_cli.visualizer = "kit" env_builder = ArenaEnvBuilder(isaaclab_arena_environment, args_cli) -# Build the env_cfg so we can inject extra event terms before registration. +# ``compose_manager_cfg`` collects every enabled variation from the scene and +# merges their event terms into ``env_cfg.events`` automatically (see +# ``ArenaEnvBuilder._compose_variations_event_cfg``). No manual plumbing. env_cfg = env_builder.compose_manager_cfg() +assert ( + env_cfg.scene.replicate_physics is False +), "Per-env color variation requires replicate_physics=False; got True." # %% -# Per-env visual tint via a custom event (see ``tint_events.py``). +# --- In-place diffuse tint (still a TODO, see 2026_04_21_color_variation_status.md) -- # -# This *keeps* the asset's original diffuse texture and multiplies a random -# color onto it, rather than replacing the material with a flat OmniPBR -# instance like ``mdp.randomize_visual_color`` does. -# -# Requirements: -# * ``scene.replicate_physics`` must be False (Arena default) — with replication -# on, every env shares a single source material and per-env tinting is -# impossible. The event itself also asserts this. -# * ``mode="prestartup"`` → each env gets a stable tint for the entire run. -# Use ``mode="reset"`` instead to resample on every episode reset. -# * The colors dict specifies uniform ranges per channel. Narrow ranges near -# 1.0 (e.g. (0.4, 1.0)) give subtle, photo-realistic tints; wide ranges -# like (0.0, 1.0) look more aggressive. -assert ( - env_cfg.scene.replicate_physics is False -), "randomize_visual_diffuse_tint requires replicate_physics=False; got True." +# The in-place tint path (``randomize_visual_diffuse_tint``) preserves the +# asset's diffuse texture but currently doesn't produce a visible change. +# Left here so it's easy to A/B once the shader-path fix lands. # env_cfg.events.cracker_box_tint = EventTermCfg( # func=randomize_visual_diffuse_tint, # mode="reset", @@ -92,36 +98,6 @@ # "colors": {"r": (0.4, 1.0), "g": (0.4, 1.0), "b": (0.4, 1.0)}, # }, # ) -# env_cfg.events.tomato_soup_can_tint = EventTermCfg( -# func=randomize_visual_diffuse_tint, -# mode="prestartup", -# params={ -# "asset_cfg": SceneEntityCfg(tomato_soup_can.name), -# "colors": {"r": (0.4, 1.0), "g": (0.4, 1.0), "b": (0.4, 1.0)}, -# }, -# ) - -# --- Previous behavior: replace the material entirely (texture is lost) --- -env_cfg.events.cracker_box_color = EventTermCfg( - func=mdp.randomize_visual_color, - mode="reset", - params={ - "asset_cfg": SceneEntityCfg(cracker_box.name), - "colors": {"r": (0.0, 1.0), "g": (0.0, 1.0), "b": (0.0, 1.0)}, - "mesh_name": "", - "event_name": "cracker_box_color", - }, -) -# env_cfg.events.tomato_soup_can_color = EventTermCfg( -# func=mdp.randomize_visual_color, -# mode="prestartup", -# params={ -# "asset_cfg": SceneEntityCfg(tomato_soup_can.name), -# "colors": [(1.0, 0.0, 0.0), (0.0, 1.0, 0.0), (0.0, 0.0, 1.0), (1.0, 1.0, 0.0)], -# "mesh_name": "", -# "event_name": "tomato_soup_can_color", -# }, -# ) # %% diff --git a/isaaclab_arena/scene/scene.py b/isaaclab_arena/scene/scene.py index 2faf83060..936109b56 100644 --- a/isaaclab_arena/scene/scene.py +++ b/isaaclab_arena/scene/scene.py @@ -13,12 +13,13 @@ from isaaclab_arena.assets.asset import Asset from isaaclab_arena.assets.object import Object -from isaaclab_arena.assets.object_base import ObjectType +from isaaclab_arena.assets.object_base import ObjectBase, ObjectType from isaaclab_arena.assets.object_reference import ObjectReference from isaaclab_arena.assets.object_set import RigidObjectSet from isaaclab_arena.environments.isaaclab_arena_manager_based_env import IsaacLabArenaManagerBasedRLEnvCfg from isaaclab_arena.utils.configclass import make_configclass from isaaclab_arena.utils.phyx_utils import add_contact_report +from isaaclab_arena.variations.variation_base import VariationBase AssetCfg = Union[AssetBaseCfg, RigidObjectCfg, ArticulationCfg, ContactSensorCfg] @@ -102,6 +103,14 @@ def get_curriculum_cfg(self) -> Any: def get_commands_cfg(self) -> Any: return self.commands_cfg + def get_variations(self) -> list[VariationBase]: + """Return all enabled variations declared on :class:`ObjectBase` assets in the scene.""" + variations: list[VariationBase] = [] + for asset in self.assets.values(): + if isinstance(asset, ObjectBase): + variations.extend(asset.get_variations()) + return variations + def get_objects_with_relations(self) -> list[Object | ObjectReference]: """Return all objects in the scene that have at least one relation.""" objects_with_relations: list[Object | ObjectReference] = [] diff --git a/isaaclab_arena/variations/object_color.py b/isaaclab_arena/variations/object_color.py index 1cc27ce5e..297f1aa14 100644 --- a/isaaclab_arena/variations/object_color.py +++ b/isaaclab_arena/variations/object_color.py @@ -33,7 +33,14 @@ from isaaclab_arena.scene.scene import Scene -@register_variation("color") +#: Default RGB sampler used when the user enables the variation without +#: calling :meth:`~isaaclab_arena.variations.variation_base.VariationBase.set_sampler`. +#: Full ``[0, 1]^3`` range — aggressive but universally valid; users wanting +#: subtler tints can override via ``set_sampler(UniformSampler(low=(0.4,)*3, high=(1.0,)*3))``. +DEFAULT_COLOR_SAMPLER = UniformSampler(low=(0.0, 0.0, 0.0), high=(1.0, 1.0, 1.0)) + + +@register_variation class ObjectColorVariation(VariationBase): """Randomize an object's visual color per env. @@ -41,7 +48,9 @@ class ObjectColorVariation(VariationBase): :class:`isaaclab.envs.mdp.randomize_visual_color`. The target asset's bound material is replaced with a fresh ``OmniPBR`` instance whose ``diffuse_color_constant`` is sampled (uniformly over RGB) from the - bounds of the sampler provided via + variation's sampler. The sampler defaults to :data:`DEFAULT_COLOR_SAMPLER` + so calling :meth:`enable` alone is sufficient for reasonable behaviour; + users can narrow or replace the distribution via :meth:`~isaaclab_arena.variations.variation_base.VariationBase.set_sampler`. Requirements: @@ -63,6 +72,8 @@ class ObjectColorVariation(VariationBase): meshes under the asset's prim. """ + name = "color" + def __init__( self, asset: ObjectBase, @@ -72,6 +83,7 @@ def __init__( super().__init__(asset) self.mode = mode self.mesh_name = mesh_name + self.set_sampler(DEFAULT_COLOR_SAMPLER) def build_event_cfg(self, scene: Scene) -> tuple[str, EventTermCfg]: # noqa: ARG002 assert self._sampler is not None, ( diff --git a/isaaclab_arena/variations/variation_base.py b/isaaclab_arena/variations/variation_base.py index 11b983dcb..486ea37b6 100644 --- a/isaaclab_arena/variations/variation_base.py +++ b/isaaclab_arena/variations/variation_base.py @@ -7,17 +7,19 @@ A :class:`VariationBase` describes *one* knob to turn on the scene — the target asset, the sampler that drives it, and the event term that realises -it. Variations are instantiated by their target asset as part of -:meth:`~isaaclab_arena.assets.object_base.ObjectBase.available_variations` -(disabled + sampler-less by default) and then configured by the user via -:meth:`enable` / :meth:`set_sampler`. The builder walks the scene, collects -enabled variations, and merges their event terms into ``events_cfg``. +it. Variations are attached to their target asset in the asset's ``__init__`` +via :meth:`~isaaclab_arena.assets.object_base.ObjectBase.add_variation` +(disabled by default, pre-configured with a sensible default sampler where +applicable) and then toggled by the user via +:meth:`enable` (and optionally narrowed via :meth:`set_sampler`). The builder +walks the scene, collects enabled variations, and merges their event terms +into ``events_cfg``. """ from __future__ import annotations from abc import ABC, abstractmethod -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, ClassVar from isaaclab.managers import EventTermCfg @@ -32,13 +34,28 @@ class VariationBase(ABC): A variation binds a target asset and a :class:`~isaaclab_arena.variations.sampler.Sampler` to the event term - required to apply it on reset / prestartup. It starts disabled with no - sampler; the user flips it on via :meth:`enable` and supplies a sampler - via :meth:`set_sampler` before the environment is built. Subclasses + required to apply it on reset / prestartup. It starts disabled; concrete + subclasses typically install a default sampler in their constructor so + the user can flip it on with a single :meth:`enable` call and override + the distribution later via :meth:`set_sampler` if desired. Subclasses implement :meth:`build_event_cfg`, which the builder calls once per enabled variation. + + Concrete subclasses must also declare a class-level :attr:`name` — a short, + unique identifier used both as the key under which the asset stores the + variation (see + :meth:`~isaaclab_arena.assets.object_base.ObjectBase.add_variation`) and as + the registry key picked up by + :func:`~isaaclab_arena.variations.variation_registry.register_variation`. """ + #: Short, unique identifier for this variation kind (e.g. ``"color"``, + #: ``"mass"``). Concrete subclasses **must** set this; abstract intermediates + #: may leave it unset. Used by + #: :class:`~isaaclab_arena.variations.variation_registry.VariationRegistry` + #: and :meth:`~isaaclab_arena.assets.object_base.ObjectBase.add_variation`. + name: ClassVar[str] + def __init__(self, asset: ObjectBase): self.asset = asset self._enabled: bool = False diff --git a/isaaclab_arena/variations/variation_registry.py b/isaaclab_arena/variations/variation_registry.py index 176babed0..32c285d90 100644 --- a/isaaclab_arena/variations/variation_registry.py +++ b/isaaclab_arena/variations/variation_registry.py @@ -53,18 +53,25 @@ def entries(cls) -> dict[str, type[VariationBase]]: return dict(cls._entries) -def register_variation(name: str): - """Decorator: register a :class:`VariationBase` subclass under ``name``. +def register_variation(cls: type[VariationBase]) -> type[VariationBase]: + """Decorator: register a :class:`VariationBase` subclass under its ``name``. + + The variation's class-level :attr:`~VariationBase.name` attribute is used + as the registry key — the same name the asset keys it under when it is + attached via + :meth:`~isaaclab_arena.assets.object_base.ObjectBase.add_variation`. This + keeps the "variation name" defined in exactly one place: the class itself. Example:: - @register_variation("color") + @register_variation class ObjectColorVariation(VariationBase): + name = "color" ... """ - - def decorator(cls: type[VariationBase]) -> type[VariationBase]: - VariationRegistry.register(name, cls) - return cls - - return decorator + assert isinstance(getattr(cls, "name", None), str) and cls.name, ( + f"Variation {cls.__module__}.{cls.__name__} must declare a non-empty " + "class-level `name: ClassVar[str]` before @register_variation." + ) + VariationRegistry.register(cls.name, cls) + return cls From fb5a6f2fbaa59f89807ee50a62c9d9631442cd3f Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 23 Apr 2026 09:58:48 +0200 Subject: [PATCH 05/25] Fixed recursive descent. --- .../environments/arena_env_builder.py | 3 ++- .../examples/compile_env_notebook.py | 17 ++++++++------ isaaclab_arena/variations/object_color.py | 9 ++++---- isaaclab_arena/variations/variation_base.py | 23 ++++++++++--------- 4 files changed, 29 insertions(+), 23 deletions(-) diff --git a/isaaclab_arena/environments/arena_env_builder.py b/isaaclab_arena/environments/arena_env_builder.py index dc5d44e7e..208404fe4 100644 --- a/isaaclab_arena/environments/arena_env_builder.py +++ b/isaaclab_arena/environments/arena_env_builder.py @@ -254,7 +254,8 @@ def compose_manager_cfg(self) -> IsaacLabArenaManagerBasedRLEnvCfg: [("placement_reset", EventTermCfg, self._placement_event_cfg)], ) placement_event_cfg = PlacementEventCfg() - variations_event_cfg = self._compose_variations_event_cfg() + # variations_event_cfg = self._compose_variations_event_cfg() + variations_event_cfg = None events_cfg = combine_configclass_instances( "EventsCfg", embodiment.get_events_cfg(), diff --git a/isaaclab_arena/examples/compile_env_notebook.py b/isaaclab_arena/examples/compile_env_notebook.py index 89ee7f414..b1d338716 100644 --- a/isaaclab_arena/examples/compile_env_notebook.py +++ b/isaaclab_arena/examples/compile_env_notebook.py @@ -25,13 +25,18 @@ # %% import isaaclab.envs.mdp as mdp # noqa: F401 (kept for the commented-out in-place tint notes below) -from isaaclab.managers import EventTermCfg, SceneEntityCfg # noqa: F401 (kept for the commented-out in-place tint notes below) +from isaaclab.managers import ( # noqa: F401 (kept for the commented-out in-place tint notes below) + EventTermCfg, + SceneEntityCfg, +) from isaaclab_arena.assets.registries import AssetRegistry from isaaclab_arena.cli.isaaclab_arena_cli import get_isaaclab_arena_cli_parser from isaaclab_arena.environments.arena_env_builder import ArenaEnvBuilder from isaaclab_arena.environments.isaaclab_arena_environment import IsaacLabArenaEnvironment -from isaaclab_arena.examples.tint_events import randomize_visual_diffuse_tint # noqa: F401 (kept for the commented-out in-place tint notes below) +from isaaclab_arena.examples.tint_events import ( # noqa: F401 (kept for the commented-out in-place tint notes below) + randomize_visual_diffuse_tint, +) from isaaclab_arena.relations.relations import IsAnchor, On from isaaclab_arena.scene.scene import Scene from isaaclab_arena.utils.pose import Pose @@ -55,10 +60,10 @@ # :meth:`~isaaclab_arena.variations.variation_base.VariationBase.enable` alone # is enough to get reasonable behaviour; :meth:`set_sampler` is only needed to # narrow or replace the default distribution. -cracker_box.get_variation("color").enable() # uses the default full-RGB sampler +# cracker_box.get_variation("color").enable() # uses the default full-RGB sampler # Uncomment to also randomize the soup can with a tighter (pastel) range: -# tomato_soup_can.get_variation("color").enable() +tomato_soup_can.get_variation("color").enable() # tomato_soup_can.get_variation("color").set_sampler(UniformSampler(low=(0.4,) * 3, high=(1.0,) * 3)) scene = Scene(assets=[background, cracker_box, tomato_soup_can]) @@ -79,9 +84,7 @@ # merges their event terms into ``env_cfg.events`` automatically (see # ``ArenaEnvBuilder._compose_variations_event_cfg``). No manual plumbing. env_cfg = env_builder.compose_manager_cfg() -assert ( - env_cfg.scene.replicate_physics is False -), "Per-env color variation requires replicate_physics=False; got True." +assert env_cfg.scene.replicate_physics is False, "Per-env color variation requires replicate_physics=False; got True." # %% diff --git a/isaaclab_arena/variations/object_color.py b/isaaclab_arena/variations/object_color.py index 297f1aa14..14c9fdf30 100644 --- a/isaaclab_arena/variations/object_color.py +++ b/isaaclab_arena/variations/object_color.py @@ -80,23 +80,24 @@ def __init__( mode: str = "reset", mesh_name: str = "", ): - super().__init__(asset) + super().__init__() + self.asset_name = asset.name self.mode = mode self.mesh_name = mesh_name self.set_sampler(DEFAULT_COLOR_SAMPLER) def build_event_cfg(self, scene: Scene) -> tuple[str, EventTermCfg]: # noqa: ARG002 assert self._sampler is not None, ( - f"ObjectColorVariation on '{self.asset.name}' is enabled but no sampler is set; " + f"ObjectColorVariation on '{self.asset_name}' is enabled but no sampler is set; " "call .set_sampler(...) before building the env." ) colors = self._sampler_to_colors_spec() - event_name = f"{self.asset.name}_color_variation" + event_name = f"{self.asset_name}_color_variation" event_cfg = EventTermCfg( func=mdp.randomize_visual_color, mode=self.mode, params={ - "asset_cfg": SceneEntityCfg(self.asset.name), + "asset_cfg": SceneEntityCfg(self.asset_name), "colors": colors, "mesh_name": self.mesh_name, "event_name": event_name, diff --git a/isaaclab_arena/variations/variation_base.py b/isaaclab_arena/variations/variation_base.py index 486ea37b6..81b1a4b44 100644 --- a/isaaclab_arena/variations/variation_base.py +++ b/isaaclab_arena/variations/variation_base.py @@ -5,15 +5,18 @@ """Variation abstract base class. -A :class:`VariationBase` describes *one* knob to turn on the scene — the -target asset, the sampler that drives it, and the event term that realises -it. Variations are attached to their target asset in the asset's ``__init__`` -via :meth:`~isaaclab_arena.assets.object_base.ObjectBase.add_variation` +A :class:`VariationBase` describes *one* knob to turn on the scene — a +sampler that drives it and the event term that realises it. Concrete +subclasses are responsible for remembering the target asset (typically by +name, not by reference, to avoid back-edges into the asset graph that can +trip reference-walking validators like +:func:`isaaclab.utils.configclass._validate`). Variations are attached to +their target asset in the asset's ``__init__`` via +:meth:`~isaaclab_arena.assets.object_base.ObjectBase.add_variation` (disabled by default, pre-configured with a sensible default sampler where -applicable) and then toggled by the user via -:meth:`enable` (and optionally narrowed via :meth:`set_sampler`). The builder -walks the scene, collects enabled variations, and merges their event terms -into ``events_cfg``. +applicable) and then toggled by the user via :meth:`enable` (and optionally +narrowed via :meth:`set_sampler`). The builder walks the scene, collects +enabled variations, and merges their event terms into ``events_cfg``. """ from __future__ import annotations @@ -24,7 +27,6 @@ from isaaclab.managers import EventTermCfg if TYPE_CHECKING: - from isaaclab_arena.assets.object_base import ObjectBase from isaaclab_arena.scene.scene import Scene from isaaclab_arena.variations.sampler import Sampler @@ -56,8 +58,7 @@ class VariationBase(ABC): #: and :meth:`~isaaclab_arena.assets.object_base.ObjectBase.add_variation`. name: ClassVar[str] - def __init__(self, asset: ObjectBase): - self.asset = asset + def __init__(self): self._enabled: bool = False self._sampler: Sampler | None = None From f519018165277fe8fb7b760729edc02c0c065f0c Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 23 Apr 2026 11:11:43 +0200 Subject: [PATCH 06/25] Different colors per object in randomize_visual_color. --- isaaclab_arena/environments/arena_env_builder.py | 3 +-- isaaclab_arena/examples/compile_env_notebook.py | 2 +- submodules/IsaacLab | 2 +- 3 files changed, 3 insertions(+), 4 deletions(-) diff --git a/isaaclab_arena/environments/arena_env_builder.py b/isaaclab_arena/environments/arena_env_builder.py index 208404fe4..dc5d44e7e 100644 --- a/isaaclab_arena/environments/arena_env_builder.py +++ b/isaaclab_arena/environments/arena_env_builder.py @@ -254,8 +254,7 @@ def compose_manager_cfg(self) -> IsaacLabArenaManagerBasedRLEnvCfg: [("placement_reset", EventTermCfg, self._placement_event_cfg)], ) placement_event_cfg = PlacementEventCfg() - # variations_event_cfg = self._compose_variations_event_cfg() - variations_event_cfg = None + variations_event_cfg = self._compose_variations_event_cfg() events_cfg = combine_configclass_instances( "EventsCfg", embodiment.get_events_cfg(), diff --git a/isaaclab_arena/examples/compile_env_notebook.py b/isaaclab_arena/examples/compile_env_notebook.py index b1d338716..ccea420fe 100644 --- a/isaaclab_arena/examples/compile_env_notebook.py +++ b/isaaclab_arena/examples/compile_env_notebook.py @@ -40,7 +40,6 @@ from isaaclab_arena.relations.relations import IsAnchor, On from isaaclab_arena.scene.scene import Scene from isaaclab_arena.utils.pose import Pose -from isaaclab_arena.variations import UniformSampler asset_registry = AssetRegistry() @@ -63,6 +62,7 @@ # cracker_box.get_variation("color").enable() # uses the default full-RGB sampler # Uncomment to also randomize the soup can with a tighter (pastel) range: +cracker_box.get_variation("color").enable() tomato_soup_can.get_variation("color").enable() # tomato_soup_can.get_variation("color").set_sampler(UniformSampler(low=(0.4,) * 3, high=(1.0,) * 3)) diff --git a/submodules/IsaacLab b/submodules/IsaacLab index e57379c63..0c25b88f0 160000 --- a/submodules/IsaacLab +++ b/submodules/IsaacLab @@ -1 +1 @@ -Subproject commit e57379c634b42db5a0fe9f754341be6e2a7c7c43 +Subproject commit 0c25b88f0f63a94fd7ebcb5edfee2bb32138834f From bab514fabbcecc1f9c5480a228d780b686a3b82e Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 23 Apr 2026 11:18:55 +0200 Subject: [PATCH 07/25] Update the plan and status. --- 2026_04_21_color_variation_status.md | 42 +++---- 2026_04_21_variation_system_plan.md | 165 +++++++++++---------------- 2 files changed, 79 insertions(+), 128 deletions(-) diff --git a/2026_04_21_color_variation_status.md b/2026_04_21_color_variation_status.md index 34d55a7ee..33dbda70f 100644 --- a/2026_04_21_color_variation_status.md +++ b/2026_04_21_color_variation_status.md @@ -1,39 +1,23 @@ # Color Variation — POC Status -Companion to [2026_04_21_variation_system_plan.md](2026_04_21_variation_system_plan.md). Tracks the state of the per-env object color randomization exploration. +Companion to [2026_04_21_variation_system_plan.md](2026_04_21_variation_system_plan.md). -## Where we are +## Status: shipped as `ObjectColorVariation` -We have a working proof-of-concept for per-env color randomization on scene objects, validated in `isaaclab_arena/examples/compile_env_notebook.py` with `num_envs=4`, the kitchen background, and two YCB objects (`cracker_box`, `tomato_soup_can`). +Per-env flat-colour randomisation works end-to-end on any `Object`, via the variation system. Validated in `isaaclab_arena/examples/compile_env_notebook.py` with `num_envs=4`, the kitchen background, and two YCB objects (`cracker_box`, `tomato_soup_can`): -**What works** — the `mdp.randomize_visual_color` event from Isaac Lab, injected post-`compose_manager_cfg()` on `env_cfg.events`. Each cloned env gets a distinct, random flat color bound to the object's top-level prim. Requires `scene.replicate_physics=False` (Arena default). +```python +cracker_box.get_variation("color").enable() +tomato_soup_can.get_variation("color").enable() +``` -**What doesn't (yet) work** — the in-place diffuse tint variant (`isaaclab_arena/examples/tint_events.py`, class `randomize_visual_diffuse_tint`). It walks the stage, finds each env's bound material shader, and writes `inputs:diffuse_tint` (MDL/OmniPBR) or `inputs:diffuseColor` (UsdPreviewSurface). The event runs without errors but the rendered objects do not visibly change color. Leaving the in-place tint code in the notebook (alongside the commented-out `randomize_visual_color` block) so we can A/B later. +Each cloned env gets a distinct random flat colour bound to the object's top-level prim. Requires `scene.replicate_physics=False` (Arena default). -## Next step: fold into the variation system +## Known limitations -The goal now is to promote this POC into a first-class `Variation` under the system being built per [2026_04_21_variation_system_plan.md](2026_04_21_variation_system_plan.md). Concretely: +- **Texture is dropped.** The event goes through `mdp.randomize_visual_color`, which replaces the bound material with a fresh `OmniPBR` whose `diffuse_color_constant` is randomised. The original diffuse texture is lost. An in-place tint path (preserving the texture) was prototyped in `isaaclab_arena/examples/tint_events.py` / `randomize_visual_diffuse_tint` — runs without error but the render doesn't change. Left in the notebook for A/B when we come back to it. +- Only `UniformSampler` over RGB is supported. Discrete palettes (`randomize_visual_color`'s list-of-tuples path) need a `DiscreteChoiceSampler`. -- **New variation class** (e.g. `ObjectColorVariation`) declared as an `available_variation` on `Object` (or a mix-in available to all USD-backed objects). -- **Sampler**: reuse `UniformSampler` for continuous RGB ranges, possibly add a `DiscreteChoiceSampler` for discrete palettes (matches the two code paths `randomize_visual_color` already supports). -- **`build_event_cfg(scene)`**: emit an `EventTermCfg(func=mdp.randomize_visual_color, mode="prestartup", params={...})` with `asset_cfg = SceneEntityCfg(object.name)` and the sampled `colors` spec. -- **Preconditions**: assert `scene.replicate_physics is False` in the variation's setup path, with a clear error message (Newton preset flips it on automatically — see `ArenaEnvBuilder.compose_manager_cfg`). -- **User API sketch**: - ```python - cracker_box.set_variation("color", UniformSampler(low=(0.0,)*3, high=(1.0,)*3)) - ``` - The builder collects these the same way it will collect `ObjectMassVariation` and merges them into `events_cfg`. +## Next -Doing this through the variation system also removes the notebook-specific `env_cfg.events.cracker_box_color = ...` plumbing — users declare the variation on the asset and the builder wires it up. - -## TODOs - -- [ ] **Get `randomize_visual_diffuse_tint` (in-place tint) working** — investigation needed on why the shader-input writes don't visibly affect the render. Candidate causes (in rough priority order): - 1. The YCB assets may have a common `/World/Looks/` material (not per-env), so writing to it doesn't diverge per env. - 2. OmniPBR's `diffuse_tint` may require the texture graph to sample through the tint — asset shaders may route the texture directly to `diffuse_color_constant` instead, making `diffuse_tint` a no-op. Writing `diffuse_color_constant` (while a texture is connected) is a cleaner knob to try next. - 3. The MDL shader input may need to be set on the material prim rather than the shader prim (check `info:mdl:sourceAsset` vs MDL parameter spec). - 4. Instanceability may have been disabled too late — the material resolution might have been cached before `SetInstanceable(False)` ran. - A useful next session is to open the cracker_box USD in Isaac Sim's stage inspector, find the actual shader path, and experiment manually in the Script Editor before re-coding. -- [ ] **Promote the color variation into `ObjectColorVariation`** per the plan above. -- [ ] Decide whether `mdp.randomize_visual_color` (replacement, texture lost) or the in-place tint (texture preserved) is the primary path. Ideally the variation system lets the user choose between them via the variation's constructor args. -- [ ] Add a regression test that spins up a small scene with `num_envs>=2` and asserts the material bindings or shader inputs differ across envs. (Rendering-based assertions would be heavier — a stage-level assertion is probably sufficient.) +- Explore how to do configurations of the variations from the command line using Hydra. diff --git a/2026_04_21_variation_system_plan.md b/2026_04_21_variation_system_plan.md index a8dee1418..c3a1acf7f 100644 --- a/2026_04_21_variation_system_plan.md +++ b/2026_04_21_variation_system_plan.md @@ -1,142 +1,109 @@ -# Variation System — Mass POC Plan +# Variation System — Plan -Companion to [2026_04_13_sensitivity_analysis.md](2026_04_13_sensitivity_analysis.md). This plan covers the first slice of the sensitivity-analysis feature: the *variation system* only (analysis tooling is deferred). +Companion to [2026_04_13_sensitivity_analysis.md](2026_04_13_sensitivity_analysis.md). First slice of the sensitivity-analysis feature: the *variation system* only. Analysis tooling is deferred. ## Goal -Build the skeleton of a variation system (samplers + variation base + registry) and validate it end-to-end with **one** concrete variation: `ObjectMassVariation` using a `UniformSampler` with hardcoded parameters. No CLI, no eval_runner config, no other variations. +Build a variation system (samplers + variation base + registry) and validate it end-to-end with one concrete variation: `ObjectColorVariation` using a `UniformSampler` over RGB. No CLI, no eval_runner config, no other variations yet. Status: shipped — see [2026_04_21_color_variation_status.md](2026_04_21_color_variation_status.md). -## Design overview +## Design -Variations are declared by **asset classes** (analogous to how `ObjectBase` already declares event terms via `get_event_cfg()`). An asset class advertises which variations it *supports*; an asset instance holds the set of variations the user has *enabled* on it by supplying a sampler. The builder collects all enabled variations scene-wide and merges their event terms into `events_cfg`. +Every `Object` subclass attaches the variations it supports in `__init__`, disabled by default. Users opt in via `get_variation(name).enable()` and optionally narrow the distribution with `set_sampler(...)`. The builder walks the scene, collects enabled variations, and merges their event terms into `events_cfg`. ```mermaid flowchart LR - Cls["Object subclass\navailable_variations = {mass: ObjectMassVariation}"] -.declares.-> Inst - User[User code] -->|"obj.set_variation('mass', UniformSampler(0.1, 1.0))"| Inst[Object instance\n_enabled_variations] + Ctor["Object.__init__\nself.add_variation(ObjectColorVariation(self))"] --> Inst[Object instance\n_variations dict] + User["cracker_box.get_variation('color').enable()"] --> Inst Inst --> Scene["Scene.get_variations()"] - EnvExtras[IsaacLabArenaEnvironment.variations\nscene-wide escape hatch] --> Builder - Scene --> Builder[ArenaEnvBuilder.compose_manager_cfg] - Builder -->|"variation.build_event_cfg(scene)"| ETC[EventTermCfg randomize_object_mass] + Scene --> Builder["ArenaEnvBuilder._compose_variations_event_cfg"] + Builder -->|"variation.build_event_cfg(scene)"| ETC[EventTermCfg] ETC --> Events[events_cfg] - Events -->|at reset, per env_ids| Fn[randomize_object_mass term fn] - Fn -->|samples| S[UniformSampler.sample num_envs] - Fn -->|writes| Sim[RigidObject.root_physx_view.set_masses] ``` Key separations: -- **Sampler**: "how to draw values" (stateless distribution object). Seeded via an RNG passed in at sample time. -- **Variation**: "what knob to turn + how to turn it". Owns a sampler and an asset target; knows how to emit an `EventTermCfg`. -- **Asset class**: declares *supported* variations (class-level capability). Does **not** enable anything by default. -- **Asset instance**: holds *enabled* variations (user-configured sampler present). Default state = no variation (deterministic). -- **Registry**: global `name → Variation class` table. Not consumed by anything in this POC but establishes the naming contract for later CLI resolution. -- **Integration**: one new step in `ArenaEnvBuilder.compose_manager_cfg` collects `scene.get_variations() + arena_env.variations` and merges their event terms into `events_cfg`. +- **Sampler**: stateless distribution (`Sampler` ABC + `UniformSampler`). RNG passed in at sample time. +- **Variation**: one knob. Owns a sampler, remembers its target asset *by name only* (see "gotchas"), emits an `EventTermCfg`. +- **Asset**: instantiates its supported variations in `__init__`, disabled. User flips them on. +- **Registry**: global `name → Variation class` table populated by `@register_variation`. Naming contract for later CLI resolution; not consumed by the builder yet. -## New module layout +## Module layout -- `isaaclab_arena/variations/__init__.py` — re-exports; triggers registrations via imports. +- `isaaclab_arena/variations/__init__.py` — public re-exports; import triggers registrations. - `isaaclab_arena/variations/sampler.py` — `Sampler` ABC + `UniformSampler`. -- `isaaclab_arena/variations/base.py` — `Variation` ABC + `VariationRegistry` + `@register_variation` decorator. -- `isaaclab_arena/variations/object_mass.py` — `ObjectMassVariation` + event term function `randomize_object_mass`. +- `isaaclab_arena/variations/variation_base.py` — `VariationBase` ABC. +- `isaaclab_arena/variations/variation_registry.py` — `VariationRegistry` + `@register_variation`. +- `isaaclab_arena/variations/object_color.py` — `ObjectColorVariation` (first concrete variation). -## Core interfaces (sketch) +## Core interfaces ```python -class Sampler(abc.ABC): - @abc.abstractmethod - def sample(self, num_samples: int, generator: torch.Generator) -> torch.Tensor: ... +class Sampler(ABC): + def sample(self, num_samples, generator=None) -> torch.Tensor: ... class UniformSampler(Sampler): - def __init__(self, low: float, high: float): ... - -class Variation(abc.ABC): - @abc.abstractmethod - def build_event_cfg(self, scene: Scene) -> dict[str, EventTermCfg]: ... - -@register_variation("mass") -class ObjectMassVariation(Variation): - def __init__(self, asset_name: str, sampler: Sampler): ... - def build_event_cfg(self, scene): ... # returns {"_mass_variation": EventTermCfg(...)} -``` - -`randomize_object_mass(env, env_ids, asset_cfg, sampler)`: mirrors Isaac Lab's `randomize_rigid_body_mass` shape but pulls values from our `Sampler` so the variation controls the RNG (see [isaaclab_arena/terms/events.py](isaaclab_arena/terms/events.py) for the existing per-env event pattern, e.g. `set_object_pose_per_env`). Writes masses via `rigid_object.root_physx_view.set_masses(...)`. - -### Asset-declared variation support - -On [isaaclab_arena/assets/object_base.py](isaaclab_arena/assets/object_base.py): - -```python -class ObjectBase: - @classmethod - def available_variations(cls) -> dict[str, type[Variation]]: - return {} # subclasses extend - - def set_variation(self, name: str, sampler: Sampler) -> None: - v_cls = type(self).available_variations()[name] # KeyError if unsupported - self._enabled_variations[name] = v_cls(asset_name=self.name, sampler=sampler) - - def get_variations(self) -> list[Variation]: - return list(self._enabled_variations.values()) + def __init__(self, low, high): ... # scalar or broadcastable sequence + +class VariationBase(ABC): + name: ClassVar[str] + def enable(self): ... + def set_sampler(self, sampler: Sampler): ... + @abstractmethod + def build_event_cfg(self, scene: Scene) -> tuple[str, EventTermCfg]: ... + +@register_variation +class ObjectColorVariation(VariationBase): + name = "color" + def __init__(self, asset: ObjectBase, mode="reset", mesh_name=""): + self.asset_name = asset.name # name only, no back-ref to the asset ``` -On [isaaclab_arena/assets/object.py](isaaclab_arena/assets/object.py) (rigid-body `Object`): +Attached from the asset's `__init__`: ```python class Object(ObjectBase): - @classmethod - def available_variations(cls): - return {**super().available_variations(), "mass": ObjectMassVariation} + def __init__(self, ...): + ... + self.add_variation(ObjectColorVariation(self)) # disabled until .enable() ``` -Every existing rigid object (e.g. `cracker_box`) picks up mass variation support for free. Articulated / non-rigid subclasses can extend the mapping with their own variations later (e.g. `joint_stiffness`). +## Gotchas we hit -`Scene.get_variations()` walks assets and concatenates their enabled variations. +- **Don't back-reference the asset on the variation.** `configclass._validate` walks `obj.__dict__` recursively with no cycle check; `Object._variations["color"].asset → Object` creates an unbounded reference cycle that explodes validation with `RecursionError`. `VariationBase` stores nothing about the asset; concrete subclasses store the asset *name*. +- **`randomize_visual_color` RNGs collide.** Upstream seeds its `ReplicatorRNG` from the global seed with no per-term entropy, so two enabled colour variations emit byte-identical colour streams. `ObjectColorVariation` wires a local `_PerEventSeededRandomizeVisualColor` subclass that re-seeds the RNG from a hash of the (unique) `event_name`. +- **`scene.replicate_physics` must be `False`** for per-env material divergence. Newton preset flips it on; the example notebook asserts against it at compose time. ## Integration touchpoints -- [isaaclab_arena/assets/object_base.py](isaaclab_arena/assets/object_base.py) — add `available_variations()`, `set_variation()`, `get_variations()`, and the `_enabled_variations` instance dict. -- [isaaclab_arena/assets/object.py](isaaclab_arena/assets/object.py) — extend `available_variations()` with `"mass": ObjectMassVariation` so every rigid object supports it. -- [isaaclab_arena/scene/scene.py](isaaclab_arena/scene/scene.py) — add `get_variations()` walking its assets. -- [isaaclab_arena/environments/isaaclab_arena_environment.py](isaaclab_arena/environments/isaaclab_arena_environment.py) — add an env-level `variations: list[Variation]` escape hatch (and a helper `add_variation`) for scene-wide variations that don't belong to any single asset. -- [isaaclab_arena/environments/arena_env_builder.py](isaaclab_arena/environments/arena_env_builder.py) — in `compose_manager_cfg`, after the existing event merges (`embodiment → scene → task → placement`), collect `scene.get_variations() + arena_env.variations` and merge each one's `build_event_cfg(scene)` into `events_cfg` using the same `combine_configclass_instances` pattern. Conflict policy for this POC: reject duplicate event term names with a clear assert. +- `isaaclab_arena/assets/object_base.py` — `add_variation`, `get_variation`, `get_variations`, `_variations` dict. +- `isaaclab_arena/assets/object.py` — `self.add_variation(ObjectColorVariation(self))` in `__init__`. +- `isaaclab_arena/scene/scene.py` — `Scene.get_variations()` walks `ObjectBase` assets. +- `isaaclab_arena/environments/arena_env_builder.py` — `_compose_variations_event_cfg()` merges enabled variations into `events_cfg` (asserts unique event names). -## Example usage (driver script/test, no CLI yet) +## Example ```python -arena_env = KitchenPickAndPlaceEnvironment(args_cli).get_env(args_cli) - -cracker_box = arena_env.scene.get_asset("cracker_box") -cracker_box.set_variation("mass", UniformSampler(0.1, 1.0)) - -env = ArenaEnvBuilder(arena_env, args_cli).make_registered() +cracker_box = asset_registry.get_asset_by_name("cracker_box")() +cracker_box.get_variation("color").enable() +# optional: cracker_box.get_variation("color").set_sampler( +# UniformSampler(low=(0.4,)*3, high=(1.0,)*3) +# ) +scene = Scene(assets=[cracker_box, ...]) ``` -## Tests - -Add [isaaclab_arena/tests/test_variations.py](isaaclab_arena/tests/test_variations.py): - -- **Unit** (no sim): `UniformSampler.sample(n)` returns shape `(n,)`, all values in `[low, high]`, reproducible under a fixed `torch.Generator` seed. -- **Unit** (no sim): `VariationRegistry` register/lookup round-trips; duplicate registration raises. -- **Unit** (no sim): `Object.available_variations()` includes `"mass"`; `set_variation("mass", sampler)` populates `get_variations()` with an `ObjectMassVariation`; `set_variation("nonexistent", ...)` raises. -- **Integration** (sim, mirrors [test_placement_events.py](isaaclab_arena/tests/test_placement_events.py) inner/outer pattern): build a minimal pick-and-place env, call `cracker_box.set_variation("mass", UniformSampler(0.1, 1.0))` *before* building, reset once, read back each env's cracker_box mass via the rigid object view, assert: (a) values lie in `[0.1, 1.0]`, (b) with `num_envs >= 2` and a fixed seed the values are not all identical. +See `isaaclab_arena/examples/compile_env_notebook.py`. -## Todos +## Open todos -- [ ] **samplers** — Add `isaaclab_arena/variations/sampler.py` with `Sampler` ABC and `UniformSampler`. -- [ ] **base** — Add `isaaclab_arena/variations/base.py` with `Variation` ABC, `VariationRegistry`, and `@register_variation` decorator. -- [ ] **mass_variation** — Add `isaaclab_arena/variations/object_mass.py` with `ObjectMassVariation` and the `randomize_object_mass` event term function. -- [ ] **asset_support** — Add `available_variations()` / `set_variation()` / `get_variations()` to `ObjectBase` and wire `mass` into the rigid-body `Object` class; have `Scene.get_variations()` collect enabled variations from assets. -- [ ] **env_field** — Add env-level `variations` list + `add_variation` helper to `IsaacLabArenaEnvironment` as an escape hatch for scene-wide variations. -- [ ] **builder_hook** — Integrate variations into `ArenaEnvBuilder.compose_manager_cfg` by merging scene+env variation event terms into `events_cfg`. -- [ ] **package_init** — Wire `isaaclab_arena/variations/__init__.py` to re-export and trigger registrations. -- [ ] **tests** — Add `isaaclab_arena/tests/test_variations.py` with sampler unit tests, registry unit tests, an `Object.set_variation` unit test, and a sim integration test for the mass variation. +- [ ] `isaaclab_arena/tests/test_variations.py` — sampler/registry unit tests + a ≥2-env sim integration test asserting per-env colour divergence. +- [ ] `ObjectMassVariation` — exercise the abstraction on a non-visual knob and a non-Replicator event path. +- [ ] `DiscreteChoiceSampler` — palette-style sampling; covers `randomize_visual_color`'s list-of-tuples path. +- [ ] Env-level variations escape hatch (scene-wide lights, HDR) — `IsaacLabArenaEnvironment.variations` + `add_variation` helper. +- [ ] CLI / Hydra plumbing for variation selection (`--variation cracker_box.color=uniform(...)`). -## Out of scope (explicit) +## Out of scope (this slice) -- CLI syntax (`--variation` flag, dotted-key parsing) — deferred. -- Other samplers (`Choose`, `Normal`) — only stub `Sampler` ABC so they slot in later. -- Other variations (pose, light, camera, HDR, object name) — deferred. -- Semantic target indirection ("pick_up_object" → asset) — use the concrete scene-entity name for now. -- Logging per-env sampled values for downstream sensitivity analysis — deferred. -- Registration-time vs run-time orchestration — this POC only handles per-env runtime variation. +- Analysis tooling / sensitivity metrics. +- Per-env sampled-value logging for downstream analysis. +- Registration-time vs run-time orchestration (POC is per-reset runtime only). +- Semantic target indirection (`"pick_up_object"` → asset). From e618564ef642463a0eabaf8639ae390b74ebb6e1 Mon Sep 17 00:00:00 2001 From: alex Date: Fri, 24 Apr 2026 18:21:55 +0200 Subject: [PATCH 08/25] Move to cfg based variataion configuration. --- 2026_04_21_variations_user_interface.py | 6 +- .../examples/compile_env_notebook.py | 36 ++- .../examples/dynamic_hydra_example.py | 5 + isaaclab_arena/examples/hydra_cli_example.py | 111 +++++++++ .../examples/hydra_dynamic_schema_example.py | 213 ++++++++++++++++++ isaaclab_arena/variations/__init__.py | 10 +- isaaclab_arena/variations/object_color.py | 94 ++++++-- isaaclab_arena/variations/sampler.py | 56 +++++ isaaclab_arena/variations/variation_base.py | 82 ++++++- submodules/IsaacLab | 2 +- 10 files changed, 570 insertions(+), 45 deletions(-) create mode 100644 isaaclab_arena/examples/dynamic_hydra_example.py create mode 100644 isaaclab_arena/examples/hydra_cli_example.py create mode 100644 isaaclab_arena/examples/hydra_dynamic_schema_example.py diff --git a/2026_04_21_variations_user_interface.py b/2026_04_21_variations_user_interface.py index 676613b71..ffc87a8f2 100644 --- a/2026_04_21_variations_user_interface.py +++ b/2026_04_21_variations_user_interface.py @@ -3,8 +3,7 @@ # # SPDX-License-Identifier: Apache-2.0 -## - +## Variations in user interface in Python # Option 1: Variations travel with asset. # DECISION: SUPPORTED @@ -25,3 +24,6 @@ asset_registry = AssetRegistry() apple = asset_registry.get_asset_by_name("apple") color_variation = ObjectColorVariation(apple, sampler=UniformSampler(low=(0.0,) * 3, high=(1.0,) * 3)) + + +## Variations in user interface in Hydra diff --git a/isaaclab_arena/examples/compile_env_notebook.py b/isaaclab_arena/examples/compile_env_notebook.py index ccea420fe..0ed2418fb 100644 --- a/isaaclab_arena/examples/compile_env_notebook.py +++ b/isaaclab_arena/examples/compile_env_notebook.py @@ -40,6 +40,7 @@ from isaaclab_arena.relations.relations import IsAnchor, On from isaaclab_arena.scene.scene import Scene from isaaclab_arena.utils.pose import Pose +from isaaclab_arena.variations import UniformSampler, UniformSamplerCfg asset_registry = AssetRegistry() @@ -52,19 +53,34 @@ cracker_box.add_relation(IsAnchor()) tomato_soup_can.add_relation(On(cracker_box)) -# --- New-style variation configuration --------------------------------------- +# --- Variation configuration -------------------------------------------------- # # Every ``Object`` ships with a registry of built-in variations (currently just # ``"color"``), pre-configured with a sensible default sampler. Calling # :meth:`~isaaclab_arena.variations.variation_base.VariationBase.enable` alone -# is enough to get reasonable behaviour; :meth:`set_sampler` is only needed to -# narrow or replace the default distribution. -# cracker_box.get_variation("color").enable() # uses the default full-RGB sampler - -# Uncomment to also randomize the soup can with a tighter (pastel) range: -cracker_box.get_variation("color").enable() -tomato_soup_can.get_variation("color").enable() -# tomato_soup_can.get_variation("color").set_sampler(UniformSampler(low=(0.4,) * 3, high=(1.0,) * 3)) +# is enough to get reasonable behaviour; ``set_sampler`` narrows or replaces +# the default distribution and accepts either a live :class:`Sampler` or a +# :class:`SamplerCfg`. +# +# Both objects randomize along a single RGB axis so the per-env tint is obvious +# at a glance: the cracker box varies red, the tomato soup can varies blue. We +# drive one via each branch of the unified ``set_sampler`` to exercise both +# surfaces side-by-side. + +# Imperative path: pass a live ``Sampler``. Convenient at code-level / in tests; +# does **not** touch ``variation.cfg``, so a Hydra / serialisation round-trip +# would miss this override. +cracker_box_color = cracker_box.get_variation("color") +cracker_box_color.set_sampler(UniformSampler(low=[0.2, 0.2, 0.0], high=[1.0, 1.0, 0.0])) +cracker_box_color.enable() + +# Declarative path: pass a ``SamplerCfg``. It is built into a live sampler +# *and* written back onto ``variation.cfg.sampler``, so the cfg stays the +# source of truth — this is the form that survives Hydra CLI overrides (see +# ``hydra_dynamic_schema_example.py``). +tomato_soup_can_color = tomato_soup_can.get_variation("color") +tomato_soup_can_color.set_sampler(UniformSamplerCfg(low=[0.0, 0.2, 0.2], high=[0.0, 1.0, 1.0])) +tomato_soup_can_color.enable() scene = Scene(assets=[background, cracker_box, tomato_soup_can]) isaaclab_arena_environment = IsaacLabArenaEnvironment( @@ -110,7 +126,7 @@ # %% # Run some zero actions. -NUM_STEPS = 500 +NUM_STEPS = 2000 for _ in tqdm.tqdm(range(NUM_STEPS)): with torch.inference_mode(): actions = torch.zeros(env.action_space.shape, device=env.unwrapped.device) diff --git a/isaaclab_arena/examples/dynamic_hydra_example.py b/isaaclab_arena/examples/dynamic_hydra_example.py new file mode 100644 index 000000000..ab4510ba6 --- /dev/null +++ b/isaaclab_arena/examples/dynamic_hydra_example.py @@ -0,0 +1,5 @@ +# Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md). +# All rights reserved. +# +# SPDX-License-Identifier: Apache-2.0 + diff --git a/isaaclab_arena/examples/hydra_cli_example.py b/isaaclab_arena/examples/hydra_cli_example.py new file mode 100644 index 000000000..b26f5b95b --- /dev/null +++ b/isaaclab_arena/examples/hydra_cli_example.py @@ -0,0 +1,111 @@ +# Copyright (c) 2025-2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md). +# All rights reserved. +# +# SPDX-License-Identifier: Apache-2.0 + +"""Minimal Hydra example for CLI parameter experimentation. + +This script has **no** Isaac Sim dependency — it is meant purely as a sandbox +for playing with Hydra's command-line override syntax before wiring it into a +real pipeline. + +Run from the container (or any env with ``hydra-core`` installed): + +.. code-block:: bash + + # 1. Print the default resolved config. + /isaac-sim/python.sh isaaclab_arena/examples/hydra_cli_example.py + + # 2. Override a leaf value using dotted paths. + /isaac-sim/python.sh isaaclab_arena/examples/hydra_cli_example.py \\ + training.lr=1e-4 training.epochs=50 + + # 3. Override a nested field and a list element. + /isaac-sim/python.sh isaaclab_arena/examples/hydra_cli_example.py \\ + model.hidden_sizes='[256,256,128]' model.activation=gelu + + # 4. Sweep (multirun): runs the script once per combination. + /isaac-sim/python.sh isaaclab_arena/examples/hydra_cli_example.py -m \\ + training.lr=1e-3,1e-4 model.activation=relu,gelu + + # 5. Show the resolved config without running the function body. + /isaac-sim/python.sh isaaclab_arena/examples/hydra_cli_example.py --cfg job + + # 6. Add a new field that isn't in the schema (Hydra rejects this by + # default with a structured config — use the ``+`` prefix to add it). + /isaac-sim/python.sh isaaclab_arena/examples/hydra_cli_example.py \\ + +experiment.note=baseline + +Hydra will create a ``outputs//