Sign Language to Speech Converter

Problem statement

People with hearing and speech impairment mainly communicate using sign language, which is not understood by most of the general population.

Abstract

The sign language to speech converter captures hand gestures using a camera and recognizes them using a trained convolutional neural network. The predicted character or word is shown on screen and converted to audible speech using a text to speech engine. This helps sign language users to communicate with non signers.

Components required

• Laptop or Raspberry Pi with camera
• USB or built in webcam
• Python with OpenCV
• Deep learning library (TensorFlow or PyTorch)
• Text to speech engine such as pyttsx3 or gTTS
• Dataset of sign language gestures
• Speakers or headphones

Block diagram

Working

A dataset of labeled gesture images is used to train the CNN model. In real time, frames from the camera are pre processed to extract the region of interest, resized and normalized. The model predicts the gesture class, which is mapped to a letter or word and displayed on screen. The text is passed to a text to speech engine which generates audio output.

Applications

• Assistive communication tool
• Learning and teaching sign language
• Interactive kiosks for basic queries
• Human computer interaction research