Booth Id:
SOFT004
Category:
Systems Software
Year:
2022
Finalist Names:
Vinchhi, Nand (School: National Public School - Koramangala)
Abstract:
Over 70 million people globally need sign language for communication. Although we see many advancements in voice-enabled technologies, there is an acute disparity between advances in these general-purpose communication interfaces (e.g. speech to text) and those designed for the speech and hearing impaired. The objective of this project was to implement a camera vision powered translator for American Sign Language, and to optimize the design and algorithms for real-time use.
Mediapipe pose estimation and Dynamic Time Warping were used to create two separate sign language translators - one for alphanumeric characters and the other for words and phrases. For characters, localized hand landmarks from a live webcam feed were compared with data extracted from reference images. If the lowest distance score is under a threshold value, a character is detected. For words and phrases, hands and upper body pose data from a live webcam feed was compared with similar data extracted from reference videos, using an algorithm called Dynamic Time Warping.
After a comprehensive evaluation of various feature extraction algorithms, It was found that localized cartesian coordinates for hands, angular values at joints for body, and the euclidean distance metric for Dynamic Time Warping yielded the highest accuracy of over 90% on a test dataset of over 150 video clips.
This novel method does not require intensive training, and has yielded a high accuracy for real-time translation. In this way, sign language translation technology that is performant, scalable and independent of external hardware was demonstrated.