An offline desktop automation system built as an Object-Oriented Programming (OOP) project. The system allows users to control Windows applications and basic system functions using voice commands and right-hand gestures, without depending on a physical mouse or keyboard.
The system combines:
- Offline Voice Recognition using Vosk
- Computer Vision using OpenCV and MediaPipe
- Gesture-Based Mouse Control
- Native Windows Automation using C++ Win32 APIs
- UDP Communication between Python and C++
- Multithreading and Shared State for parallel voice and vision processing
- Completely Offline Operation after the required dependencies and Vosk model are installed
The Python side handles voice processing, camera input, hand tracking, and gesture detection. The C++ backend receives commands through UDP Port 5005 and executes Windows-level operations.
Users can launch applications, navigate folders, control windows, and perform system operations using predefined voice commands.
The webcam tracks the user's right hand and converts specific finger positions into mouse movement and click events.
The project uses the locally stored Vosk English model, so voice commands do not require an internet connection.
Python manages computer vision and speech processing, while C++ performs native Windows automation through Win32 APIs.
Voice and vision systems operate in parallel using separate threads and communicate through a shared state.
The system recognizes multiple phrases for common operations.
- Settings:
settings,set up,configuration - Task Manager:
task manager,manager,start manager - This PC:
folders,storage,this pc,my pc - Calculator:
calculator,open calculator - Camera:
camera,open camera,web cam - Clock:
clock,time,showtime - Lock PC:
lock computer,security,protect system - Shutdown:
shut down,close pc,shut down computer - Restart:
restart,reboot
- Notepad:
start editor,editor,notes,open notes - Microsoft Word:
open word,microsoft word,start word,documents - Visual Studio:
visual studio,open vs,vs,studio - Code::Blocks:
code block,cb,cpp,c plus plus - Paint:
sketch,draw - WhatsApp:
what's up,messages,start messages - Browser:
google,open google,edge,open edge
- Videos:
videos,open videos - Music:
music,songs - Downloads:
downloads - YouTube:
entertainment,movies,fun
- Close:
close,close application,remove - Refresh:
update,five - Screenshot:
screenshot,screen capture,take screenshot - Exit Program:
exit,exit program,stop program
- Activate Mouse/Vision:
switch,activate,vision
Use your right hand in front of the webcam.
Gesture: Raise your index and middle fingers while keeping your thumb straight.
Action: Move your hand in the desired direction to control the mouse cursor.
Image:
Gesture: Keep your index and middle fingers up and close your thumb.
Action: Stops the cursor from following your hand.
Gesture: Keep your middle finger up, bend your index finger, and close your thumb. Keep the remaining fingers down.
Action: Performs a left mouse click.
Image:
Gesture: Keep your index finger up and put your middle finger down, and close your thumb. Keep the remaining fingers down.
Action: Performs a right mouse click.
Image:
Gesture: Keep both your index and middle fingers down, and close your thumb. Keep the remaining fingers down.
Action: Performs a double left click.
- Windows OS
- C++ compiler with
g++ - Python 3.x
- Webcam
- Microphone
- Internet connection only for initial dependency installation
pip install opencv-python mediapipe pyaudio numpy pyautogui vosk
The Vosk model must remain inside the project directory:
vosk-model-small-en-us-0.15/
Because the project uses a Python + C++ hybrid architecture, both components must run simultaneously.
Open a terminal in the project directory and run cpp code file Application_Commands_Executioner. The C++ server will start listening on UDP Port: 5005.
Open another terminal in the same project directory and run py file Code_Running _Body. The Python engine will initialize the voice and vision processing threads.
pip install opencv-python mediapipe pyaudio numpy pyautogui vosk
During the first execution of .exe file, Windows may display a Windows Security Alert.
If prompted:
- Allow access for the application.
- Enable Private networks.
- Click Allow access.
The application uses UDP communication on Port 5005 to communicate between the Python and C++ components.
- Keep the C++ backend running before starting the Python engine.
- Keep the Vosk model in the correct project directory.
- Make sure the webcam and microphone are available.
- Use the gestures clearly and within the camera's field of view.
- The system is designed for Windows because the C++ backend uses Win32 functionality.
- The system is designed to operate offline during normal use; no cloud speech-recognition service is required.