A gesture-controlled computer vision application built as a Object-Oriented Programming (OOP) project. This software serves as a hardware-free backup utility, allowing users to control system navigation and input commands through a laptop webcam.
- Computer Vision Tracking: Processes real-time webcam frames to track hand gestures for seamless mouse cursor movement and click events.
- Hybrid System Logic: Bridges Python vision pipelines with C++ to inject system-wide numeric commands (0-9) across any active text editor software.
- OOP Architecture: Designed with highly modular Python classes to encapsulate gesture tracking data using strict inheritance patterns.
Because this project uses a hybrid architecture, you must run the Python tracking module and the C++ system command listener in two separate terminal windows simultaneously.
-
Navigate into the project folder.
-
Install core dependencies:
pip install opencv-python mediapipe
-
Execute the core computer vision tracking script:
python PYTHON_Main_Body.py
-
Open a brand-new terminal tab or window.
-
Navigate into the same project folder.
-
Step 1: Create the Executable File Run the compilation command below to generate your own executable application from the C++ source files. You can change YourCustomName.exe` to any name you prefer for your program file:
g++ CPP_Main_Body.cpp Keyboard_KeyReceiver.cpp -o YourCustomName.exe -lws2_32
-
Step 2: Verify File Creation
Check your local project folder directory. You will see that a brand-new, compiled binary file matching your chosen name (e.g.,YourCustomName.exe) has been successfully created. -
Step 3: Run the Created Executable
Execute the generated program file to start the system-wide key listener:.\YourCustomName.exe
Windows Firewall
During the first execution of .exe file, Windows may display a Windows Security Alert.
If prompted:
- Allow access for the application.
- Enable Private networks.
- Click Allow access.
This section explains how to use the AI Virtual Mouse & Keyboard through hand gestures. Make sure your webcam is active and keep your hand clearly visible inside the camera frame.
While using the Virtual Mouse and Keyboard you can use you can use one at a time so toggling between Mouse and Keyboard use this gesture:
Gesture: Cross the wrists of your both arms.
Action: Toggling Between Mouse and Keyboard
Image:
Use your right hand for mouse operations.
Gesture: Raise your index and middle fingers while keeping your thumb straight.
Action: Move your hand in the desired direction to control the mouse cursor.
Image:
Gesture: Keep your index and middle fingers up and close your thumb.
Action: Stops the cursor from following your hand.
Gesture: Keep your middle finger up, bend your index finger, and close your thumb. Keep the remaining fingers down.
Action: Performs a left mouse click.
Image:
Gesture: Keep your index finger up and put your middle finger down, and close your thumb. Keep the remaining fingers down.
Action: Performs a right mouse click.
Image:
Gesture: Keep both your index and middle fingers down, and close your thumb. Keep the remaining fingers down.
Action: Performs a double left click.
The keyboard mode allows you to type numbers from 0–9 and use Backspace through hand gestures you can type on any application that allow to write text as What's App, MS word, Notepad etc.
Use the hand positions shown below to generate the corresponding keyboard input.
Action: Types the number 0.
Image:
Action: Types the number 1.
Image:
Action: Types the number 2.
Image:
Action: Types the number 3.
Image:
Action: Types the number 4.
Image:
Action: Types the number 5.
Image:
Action: Types the number 6.
Image:
Action: Types the number 7.
Image:
Action: Types the number 8.
Image:
Action: Types the number 9.
Image:
Action: Deletes the previous character using the Backspace key.
Image:
