Home

By Miranda Drummond, Maud Helen Hovland and Samuel Ostweg

Abstract

The Computer Vision Map Navigator enables real-time map navigation using simple hand gestures. It is designed to work seamlessly with just a single RGB camera, making it accessible and easy to use. The system supports both a MediaPipe model and a custom Keras model for gesture recognition, offering robustness and flexibility in different scenarios.

Contents

• Project Requirements
• Methodology
• System Design
• Implementation
• Testing and Evaluation
• Results
• Conclusion
• Contributions
• Appendices

Project Requirements

• Control Mechanism: Insights into the single-hand control for pan and zoom and real-time interaction capabilities.
• Evaluation Criteria: Criteria including use of a single RGB camera, application of open source tools, ease of use, and performance metrics.

Methodology

• Utilized Python as the primary development language for its extensive support of machine learning and computer vision libraries.
• Chose an RGB camera for input due to its widespread availability in consumer devices, enabling broader accessibility.
• Incorporated open-source tools and libraries including MediaPipe for real-time hand tracking and gesture recognition, and Keras for building a custom LSTM model tailored to gesture recognition tasks.
• Emphasized design considerations for ease of use, aiming for a system that requires no prior practice or learning for the user.
• The development process involved setting up a development environment, implementing gesture recognition algorithms using both MediaPipe and Keras models, and integrating these with map navigation functionalities using pyautogui for controlling Google Maps.
• Addressed challenges like varying lighting conditions, different hand sizes, and the seamless integration of gesture recognition with map navigation.

System Design

• The system architecture includes a gesture recognition system built on two models: MediaPipe for general hand tracking and a custom Keras LSTM model for advanced gesture recognition.
• Map navigation integration was achieved by translating recognized gestures into corresponding navigation commands (pan, zoom, click) using pyautogui.
• Designed a user interface that visually represents recognized gestures and the corresponding navigation action on the map, offering real-time feedback to the user.

Implementation

• The coding process involved Python scripts (app.py for MediaPipe model and keras_app.py for Keras model) with detailed documentation on setup and execution.
• Key code segments include the implementation of hand gesture recognition algorithms, integration with pyautogui for map control, and methods for processing and classifying hand landmarks.
• Demonstrates how the system meets project requirements through the successful interpretation of hand gestures for map navigation, supported by both models.

Testing and Evaluation

• Adopted a mixed-methods testing methodology encompassing unit tests for individual components and user testing to assess overall system usability and performance.
• User testing scenarios were crafted to cover a broad range of interactions, from basic map navigation to more complex gestures for zooming and panning.
• The system was evaluated by having various people test it, showing high usability and performance, with feedback leading to iterative improvements.

Results

• The system demonstrated effective map navigation control through hand gestures, with high accuracy in gesture recognition and responsiveness in real-time interaction.
• Achievements include a user-friendly interface requiring no prior learning, and flexibility in using either the MediaPipe or Keras model based on user preference or specific requirements.
• Identified limitations include dependency on lighting conditions and camera quality, with future work suggested in enhancing gesture recognition accuracy under varied conditions.

Conclusion

• The project successfully developed a Computer Vision Map Navigator that allows for intuitive, real-time map navigation using hand gestures, meeting all stated objectives.
• Reflects on the learning experience, highlighting the importance of integrating open-source tools and custom model development for solving practical computer vision tasks.
• Discusses the potential impact of such technologies in making digital navigation more accessible and interactive.

Contributions

• Miranda: End to end implementation of Keras Model, including model adaptation and syncing controls to pyautogui and then to a google maps link. Writing of Keras model documentation and wiki.
• Samuel: Development of first draft of basic hand navigation (Index finger left = left scroll, Index finger right = right scroll, Index finger up = upwards scroll) using google landmark & pyautogui. Also brought contributions for potential join illustration in camera.
• Maud: implemented the full landmark model with help from intial logic from Samuel and connected problem to the execution in terms of intuitive hand signals.

Appendices

Link to the original model which was modified for the keras model implementation.