- VisionPilot: Autonomous Driving Simulation, Computer Vision & Real-Time Perception (BeamNG.tech)
A modular Python project for autonomous driving research and prototyping, fully integrated with the BeamNG.tech simulator and Foxglove visualization. This system combines traditional computer vision and state-of-the-art deep learning (CNN, U-Net, YOLO, SCNN) with real-time sensor fusion and autonomous vehicle control to tackle:
- Lane detection (Traditional CV & SCNN)
- Traffic sign classification & detection (CNN, YOLO)
- Traffic light detection & classification (YOLO, CNN)
- Vehicle & pedestrian detection and recognition (YOLO)
- Multi-sensor fusion (Camera, LiDAR, Radar, GPS, IMU)
- Multi-model inference, real-time simulation, autonomous driving with PID control (BeamNG.tech)
- Cruise control
- Real-time visualization and monitoring (Foxglove WebSocket)
- Modular configuration system (YAML-based)
- Drive logging and telemetry
Watch the Emergency Braking System (AEB) in action with real-time radar filtering and collision avoidance:
Extended Demo: Watch the full video here
This demo shows real-time traffic sign detection and classification:
Extended Demo: Watch the full video here
VisionPilot does not yet support multi-camera. This is for demonstration purposes only.
This demo shows real-time traffic light detection and classification:
No extended Demo avaliable yet.
Watch the improved autonomous lane keeping demo (v2) in BeamNG.tech, featuring smoother fused CV+SCNN lane detection, stable PID steering, and robust cruise control:
Extended Demo: Watch the full video here
Note: Very low-light (tunnel) scenarios are not yet supported.
The original demo is still available for reference:
Lane Keeping & Multi-Model Detection Demo (v1)
See real-time LiDAR point cloud streaming and autonomous vehicle telemetry in Foxglove Studio:
Extended Demo: Watch the full video here
See real-time image segmentation using front and rear cameras:
Extended Demo: Watch the full video here
More demo videos and visualizations will be added as features are completed.
The vehicle is equipped with a comprehensive multi-sensor suite for autonomous perception and control:
| Sensor | Specification | Purpose | | --------------------------- | ---------------------------------------------------- | --------------------------------------------------------------- | --- | | Front Camera | 1920x1080 @ 50Hz, 70° FOV, Depth enabled | Lane detection, traffic signs, traffic lights, object detection | | LiDAR (Top) | 80 vertical lines, 360° horizontal, 120m range, 20Hz | Obstacle detection, 3D scene understanding | | Front Radar | 200m range, 128×64 bins, 50Hz | Collision avoidance, adaptive cruise control | | Rear Left & Right Radar | 30m range, 64×32 bins, 50Hz | Blindspot monitoring, rear object detection | | | Dual GPS | Front & rear positioning @ 50Hz | Localization reference | | IMU | 100Hz update rate | Vehicle dynamics, motion estimation |
 |
 |
 |
| Sensor Array | Front Radar | Lidar Visualization |
Configuration files are located in the
/configdirectory:
- Sign classification & Detection (CNN / YOLOv11m)
- Traffic light classification & Detection (CNN / YOLOv11m)
- Lane detection Fusion (SCNN / CV)
- Advanced lane detection using OpenCV (robust highway, lighting, outlier handling)
- Integrate Majority Voting system for CV
- ⭐ Semantic Segmentatation (Already built not implemented here yet)
- ⭐ Real-Time Object Detection (Cars, Trucks, Buses, Pedestrians, Cyclists) (Trained)
- 🔥 Speed Estimation using detection from camera and lidar
- Potentially use Multiple Object Tracking (MOT) for better speed estimation
- Pedestrian intent prediction (crossing, standing, walking along road)
- Note: Would Have to be tested in Carla as BeamNG.tech does not have pedestrians implemented
- Vehicle State Classification (Break Lights, Turn Signals, Reverse Lights)
- 🔥 Handle dashed lines better in lane detection
- 🔥 Lidar Object Detection 3D
- Detect multiple lanes
- 💤 Multi Camera Setup (Will implement after all other camera-based features are finished)
- 💤 Overtaking, Merging (Will be part of Path Planning)
- 🔥 Kalman Filtering
- Integrate Radar
- Integrate Lidar
- Integrate GPS
- Integrate IMU
- Ultrasonic Sensor Integration
- Note: Can easily be implemented with prebuilt Beamng ADAS module
- Map Matching algorithm
- 💤 SLAM (simultaneous localization and mapping)
- Build HD Map from Scratch
- Localize Vehicle on HD Map
- Sensor Health Monitoring & Redundancy
- Redundant Front Radar for AEB
- Sensor status diagnostics and failover
- Integrate vehicle control (Throttle, Steering, Braking Implemented) (PID needs further tuning)
- Integrate PIDF controller
- ⭐ Adaptive Cruise Control (Currently only basic Cruise Control implemented)
- ⭐ Automatic Emergency Braking AEB
- Support using Camera and Lidar detections
- Trajectory Predcition for surrounding vehicles
- Blindspot Monitoring (Using left/right rear short range radars)
- Traffic Rule Enforcement (Stop at red lights, stop signs, yield signs)
- Dynamic Target Speed based on Speed Limit Signs
- Global Path planning
- Local Path planning
- Lane Change Logic
- Parking Logic (Path finding / Parallel or Perpendicular)
- 💤 End-to-end driving policy learning (RL, imitation learning)
- 💤💤 Advanced traffic participant prediction (trajectory, intent)
- Integrate and test in BeamNG.tech simulation (replacing CARLA)
- Modularize and clean up BeamNG.tech pipeline
- Tweak lane detection parameters and thresholds
- Fog Weather conditions (Rain or snow not supported in BeamNG.tech)
- Traffic scenarios: driving in heavy, moderate, and light traffic
- Test all Systems in different lighting conditions (Day, Night, Dawn/Dusk, Tunnel)
- 💤💤 Test using actual RC car
- ⭐ Full Foxglove visualization integration (Overhaul needed)
- Modular YAML configuration system
- Real-time drive logging and telemetry
- Real time Annotations Overlay in Foxglove
- Show predicted trajectories in Foxglove
- Show Global and local path plans in Foxglove
- Live Map Visualization
- Containerize Models for easy deployment and scalability (Also eliminates dependency issues)
- Message Broker (redis/rabbitmq)
- Create docker compose
- Aggregator service
- Refactor beamng.py
- Add demo images and videos to README
- Add performance benchmarks section
- Add Table of Contents for easier navigation
- Vibe-Code a website for the project
- Redo project structure for better modularity
Driver Monitoring System would've been pretty cool but human drivers are not implemented in BeamNG.tech
🔥 = High Priority
⭐ = Complete but still being improved/tuned/changed (not final version)
💤 = Minimal Priority, can be addressed later
💤💤 = Very Low Priority, may not be implement
Status: This project is currently in active development. A stable, production-ready release with pre-trained models and complete documentation will be available eventually.
- Tunnel/Low-Light Scenarios: Camera depth perception fails below certain lighting thresholds
- Multi-Camera Support: Single front-facing camera only (future roadmap)
- Dashed Lane Detection: Requires improvement for better accuracy
- PID Controller Tuning: May oscillate on aggressive maneuvers
- Real-World Testing: Only validated in simulation (BeamNG.tech), for now...
- Rain/snow physics not supported in BeamNG.tech
- No native ROS2 support (custom bridge required)
- Pedestrians
- Human Drivers
Datasets:
- CU Lane, LISA, GTSRB, Mapillary, BDD100K
Simulation & Tools:
- BeamNG.tech by BeamNG GmbH
- Foxglove Studio for visualization
Special Thanks:
- Kaggle for free GPU resources (model training)
- Mr. Pratt (teacher/supervisor) for guidance
If you use VisionPilot in your research, please cite:
@software{visionpilot2025,
title={VisionPilot: Autonomous Driving Simulation, Computer Vision & Real-Time Perception},
author={Julian Stamm},
year={2025},
url={https://github.com/visionpilot-project/VisionPilot}
}Title: BeamNG.tech
Author: BeamNG GmbH
Address: Bremen, Germany
Year: 2025
Version: 0.35.0.0
URL: https://www.beamng.tech/
This project is licensed under the MIT License - see LICENSE file for details.