Full Duplex Communication System for Hearing-Impaired Users

Overview

A real-time bi-directional communication system bridging hearing-impaired individuals and the general public. The system translates spoken English into Indian Sign Language (ISL) gestures visually, while simultaneously converting ISL gestures captured from live video back into audible speech — a true full-duplex loop. This work was published in a peer-reviewed journal.

The Problem

A shortage of qualified ISL interpreters and limited public awareness create daily communication barriers for hearing-impaired individuals, leading to social isolation and reduced access to information. This project provides a low-cost, accessible technological alternative.

System Architecture

• Speech → Sign Module: Microphone input → Google Speech Recognition API → text → display of corresponding ISL gestures.
• Sign → Speech Module: Webcam capture → OpenCV gesture preprocessing → CNN (TensorFlow & Keras) gesture classification → gTTS audio output.

Key Technologies: Python · TensorFlow & Keras · OpenCV · Google Speech Recognition API · gTTS

Results

• ≈99% accuracy on gesture recognition validation dataset.
• Functional real-time prototype with both modules integrated in a closed communication loop.
• Published in the Journal of Computer Science and Engineering, Vol. 22, Issue 3, 2021.