Pyspotobserver refactor#7
Open
fmz wants to merge 3 commits into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces an ONNX-based “vision pipeline” into the PySpotObserver camera streaming flow, alongside a few streaming/config tweaks and example CLI updates to optionally run the pipeline when fetching frames.
Changes:
- Added a new
vision_pipeline.pymodule that loads an ONNX model and runs batched inference on RGB + depth inputs. - Extended
SpotCamStream.get_current_images()/async_get_current_images()with arun_pipelineflag and updated example scripts to use it. - Adjusted streaming configuration defaults/examples and updated RGB request parameters and color-correction handling.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| PySpotObserver/requirements.txt | Adds onnxruntime-gpu dependency. |
| PySpotObserver/pyspotobserver/vision_pipeline.py | New ONNX Runtime pipeline with global session/buffers. |
| PySpotObserver/pyspotobserver/config.py | Updates default request timeout. |
| PySpotObserver/pyspotobserver/camera_stream.py | Adds pipeline integration, tweaks image request params, refactors CCM application path. |
| PySpotObserver/examples/config_example.yaml | Updates example request timeout. |
| PySpotObserver/examples/basic_streaming.py | Adds --vision-pipeline option and RGB display conversion. |
| PySpotObserver/basic_streaming_copy.py | Adds an additional “unified streaming” example script. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+259
to
+264
| depth = [img.copy() for img in depth] | ||
|
|
||
| if run_pipeline and self._vision_pipeline: | ||
| return run_vision_pipeline(rgb, depth) | ||
|
|
||
| return rgb, depth |
Comment on lines
+283
to
+289
| rgb = [img.copy() for img in rgb] | ||
| depth = [img.copy() for img in depth] | ||
|
|
||
| if run_pipeline and self._vision_pipeline: | ||
| return run_vision_pipeline(rgb, depth) | ||
|
|
||
| return rgb, depth |
Comment on lines
314
to
+326
| loop = asyncio.get_event_loop() | ||
| return await loop.run_in_executor( | ||
| None, self.get_current_images, timeout, copy | ||
| ) | ||
|
|
||
| if not run_pipeline: | ||
| return await loop.run_in_executor( | ||
| None, self.get_current_images, timeout, copy, False | ||
| ) | ||
|
|
||
| # Run full pipeline in executor | ||
| def _get_and_process(): | ||
| rgb, depth = self.get_current_images(timeout, copy, False) | ||
| return run_vision_pipeline(rgb, depth) | ||
|
|
||
| return await loop.run_in_executor(None, _get_and_process) |
Comment on lines
18
to
+21
| from .config import SpotConfig, CameraType | ||
| from .color_correction import _ROBOT_CCMS | ||
| from .vision_pipeline import run_vision_pipeline | ||
|
|
Comment on lines
724
to
728
| np.multiply(img, 1.0 / 255.0, out=out_array, casting="unsafe") | ||
| if ccm is not None: | ||
| img = img @ ccm.T | ||
| np.clip(img, 0.0, 1.0, out=img) | ||
| return |
Comment on lines
+21
to
+27
| if _session is None: | ||
| ort.preload_dlls() | ||
|
|
||
| _session = ort.InferenceSession( | ||
| "C:/Users/jtoribio/Documents/aanya/fileexchange/promptda-vitl-rotated_optimized_batch2_fp16.onnx", | ||
| providers=["CUDAExecutionProvider"], | ||
| ) |
Comment on lines
68
to
71
| """JPEG quality for RGB images (0-100)""" | ||
|
|
||
| request_timeout_seconds: float = 5.0 | ||
| request_timeout_seconds: float = 500.0 | ||
| """Timeout for image requests""" |
Comment on lines
+12
to
+13
| mypy>=1.5.0 # Type checking | ||
| onnxruntime-gpu>=1.25.0 No newline at end of file |
Comment on lines
+169
to
+173
|
|
||
| depth_normalized = cv2.normalize(depth, None, 0, 255, cv2.NORM_MINMAX).astype(np.uint8) | ||
| depth_colored = cv2.applyColorMap(depth_normalized[0], cv2.COLORMAP_JET) | ||
| cv2.imshow(f"{window_prefix} - {camera_name} - RGB", rgb_display) | ||
| cv2.imshow(f"{window_prefix} - {camera_name} - Depth", depth_colored) |
Comment on lines
+261
to
+264
| if run_pipeline and self._vision_pipeline: | ||
| return run_vision_pipeline(rgb, depth) | ||
|
|
||
| return rgb, depth |
fmz
force-pushed
the
pyspotobserver-refactor
branch
from
8e3f37b to
711f5c6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.