A real-time, zero-shot instance segmentation platform built on Ultralytics YOLOE-26. Detect and segment arbitrary object categories from natural-language text prompts — no retraining required.
- Open-vocabulary detection — Recognize any category described in text (e.g.
"person, cup, laptop"), no labeled data needed. - Zero-shot instance segmentation — Pixel-level masks for unseen objects out of the box.
- Real-time inference — Live camera, image, and video processing with on-screen FPS monitoring.
- Interactive prompts — Change the detection vocabulary at runtime (press
Pin the CLI app). - GUI + CLI — A PyQt5 desktop interface and lightweight command-line entry points.
- Bilingual labels — Optional Chinese label rendering with automatic font detection.
- Modular design — Clean
src/package (camera,inferencer,visualizer,utils) that is easy to embed or extend.
- Python 3.8+
- OpenCV 4.x
- PyTorch 1.13+
- Ultralytics 8.2.0+
- PyQt5 (only for the GUI)
- A CUDA-capable GPU is optional but strongly recommended for real-time performance.
git clone https://github.com/<your-username>/open-vocabulary-instance-seg.git
cd open-vocabulary-instance-seg
pip install -r requirements.txtDownload the YOLOE-26 segmentation weights (yoloe-26s-seg-pf.pt) from the
Ultralytics YOLOE release and place it in models/.
See models/README.md for details.
python test_environment.py# Real-time camera inference (CLI)
python main.py
# Image inference
python demo_image.py
# Video inference
python demo_video.py
# Desktop GUI (PyQt5)
python gui_main.pyCLI hotkeys (main.py):
| Key | Action |
|---|---|
ESC |
Quit |
SPACE |
Pause / resume |
S |
Save current frame |
R |
Reset prompt |
P |
Enter a new prompt |
open-vocabulary-instance-seg/
├── main.py # CLI entry — real-time camera inference
├── demo_image.py # Image inference demo
├── demo_video.py # Video inference demo
├── gui_main.py # PyQt5 GUI entry point
├── config.py # Central configuration
├── test_environment.py # Environment / dependency checker
├── requirements.txt # Python dependencies
├── run.bat # Windows quick-launch menu
├── models/ # YOLOE-26 weights (see models/README.md)
│ └── README.md
├── src/ # Source package
│ ├── __init__.py
│ ├── camera.py # Camera capture module
│ ├── inferencer.py # YOLOE inference engine
│ ├── visualizer.py # Annotation / mask rendering (+ Chinese labels)
│ ├── utils.py # FPS counter, frame saving, helpers
│ ├── gui.py # Basic PyQt5 GUI
│ ├── gui_simple.py # Single-thread PyQt5 GUI (QTimer)
│ └── gui_full.py # Full-featured PyQt5 GUI
├── assets/ # Test media & references
│ ├── images/
│ ├── videos/
│ └── references/
├── outputs/ # Saved results
└── docs/ # Documentation
├── usage_guide.md
└── user_manual.md
Edit config.py to customize behavior:
# Model
MODEL_PATH = "models/yoloe-26s-seg-pf.pt"
DEVICE = "cuda" # "cpu" or "cuda"
# Inference
CONF_THRESH = 0.25 # confidence threshold
IOU_THRESH = 0.7 # NMS IoU threshold
IMG_SIZE = 640 # input resolution
# Open-vocabulary prompt (comma-separated, editable at runtime)
DEFAULT_PROMPT = "person, dog, cat, cup, chair, bag, laptop, bottle"
# Display
USE_CHINESE = False # Chinese label rendering (slower)
FONT_PATH = "" # empty = auto-detect system font
# Performance
USE_AMP = True # mixed-precision inference
WARMUP = True # model warm-up on startupfrom src.inferencer import YOLOEInferencer
from src.visualizer import Visualizer
inferencer = YOLOEInferencer(model_path="models/yoloe-26s-seg-pf.pt", device="cuda")
visualizer = Visualizer(use_chinese=False)
results = inferencer.predict(frame, prompt="person, laptop, cup")
annotated = visualizer.draw_results(frame, results, show_labels=True, show_conf=True)inferencer.set_prompt("car, bicycle, traffic light")
results = inferencer.predict(frame)DEVICE = "cuda" # GPU acceleration
IMG_SIZE = 480 # smaller input → faster
CONF_THRESH = 0.4 # fewer detections → faster post-processingThis platform wraps YOLOE ("You Only Look at Everything"), Ultralytics' open-vocabulary detection/segmentation model. It ships with the Prompt-Free (PF) variant, which uses a built-in visual prompt embedding, plus support for text-prompt models where you supply the category names at runtime. Because it does not require task-specific training, the same weights can segment entirely new object categories described on the fly.
| Platform | Image Size | Approx. FPS |
|---|---|---|
| CPU (Intel i7) | 640×640 | 10–15 |
| GPU (RTX 3060) | 640×640 | 40–60 |
| GPU (RTX 4090) | 640×640 | 80–120 |
Actual performance depends on scene complexity and the number of detected objects.
- Smart surveillance & anomaly detection
- Retail analytics (product / foot-traffic counting)
- Industrial defect inspection
- Autonomous-driving scene understanding
- Robotics perception & manipulation
- Automatic content annotation
- Multi-object tracking
- Web / REST API interface
- Result export to COCO/JSON
- TensorRT / ONNX acceleration
- INT8 quantization
Project code is released under the MIT License — see LICENSE. The underlying YOLOE model from Ultralytics is distributed under the AGPL-3.0 license; for commercial use please obtain an enterprise license from Ultralytics.
- Ultralytics YOLOE — the open-vocabulary model.
- PyTorch & OpenCV — the runtime foundations.
基于 Ultralytics YOLOE-26 的实时开放词汇实例分割平台，支持通过文本提示词零样本检测与分割任意类别物体，无需重新训练。提供命令行（摄像头 / 图片 / 视频）与 PyQt5 图形界面两种使用方式，支持中英文标签显示与实时性能监控，代码结构模块化、易于二次开发与集成。