Hand Gesture-Based Human–Computer Interaction System

Problem statement

Conventional interaction with computers relies heavily on keyboard, mouse, or touch interfaces, which are not always convenient or accessible for all users. In scenarios like presentations, lab demos, or for users with mobility limitations, contactless interaction through hand gestures can provide a more natural and inclusive way to control devices.

Abstract

This project implements a real-time hand gesture recognition system using a webcam and computer vision techniques. The system detects the user’s hand region, extracts keypoints or contour features, and classifies specific static or dynamic gestures using a trained machine learning or deep learning model. Each recognized gesture is mapped to an action such as next/previous slide, volume up/down, play/pause or launching applications. The project demonstrates a low-cost, software-based Human–Computer Interaction (HCI) approach that can be used in classrooms, smart homes, or accessibility tools.

Components required

• Laptop/PC with webcam
• Python with OpenCV
• Mediapipe or custom hand landmark detector (optional)
• Machine learning / deep learning library (Scikit-Learn or TensorFlow/Keras)
• GUI or background script for mapping gestures to system actions

Block diagram

Working

The webcam continuously captures frames of the user’s hand. The hand region is detected using color segmentation, background subtraction, or a pre-trained hand detector (e.g., Mediapipe Hands). From this region, features such as contour shape, convex hull defects, or 3D landmarks of fingers are extracted. These features are passed into a trained classifier that recognizes gestures like open palm, fist, swipe left/right, thumbs up, etc. Each gesture is bound to an OS-level action, such as simulating key presses for slide navigation or controlling media playback. The system runs in real time and can display an on-screen overlay showing the currently detected gesture.

Applications

• Touchless control for presentations in classrooms and seminars
• Accessibility interface for users with limited mobility
• Gesture-controlled media center or smart TV
• Interactive kiosks and exhibition demos
• Foundation for AR/VR gesture interaction research