Booth Id:
TECA010
Category:
Technology Enhances the Arts
Year:
2025
Finalist Names:
Spiride, Ana (School: Plano East Senior High School)
Abstract:
Nearly 5% of the global population, or 430M people, require hearing loss rehabilitation, while in the U.S., about 11M people experience serious hearing difficulties. Many deaf individuals communicate through sign language, but few hearing people understand it, making everyday interactions challenging. This project introduces an affordable, portable, two-way translator between American Sign Language (ASL) and English that uses computer vision techniques for highly accurate fast interpretation. The HandTalk system translates ASL into English through a hand pose estimation neural network and k-Nearest-Neighbors classifier, identifying ASL letters based on finger joint angles. This system runs on various platforms, including Jetson Nano computers, laptops, and tablets. Additionally, a 3D-printed, servo-actuated robot hand uses the HandTalk algorithm in reverse to translate English text into ASL signs, creating a complete ASL-to-English and English-to-ASL translation system. This setup is affordable (under $200) and adaptable to different platforms. The system's performance was tested under real-world conditions, such as varying lighting and backgrounds, with experimental groups assessing the ASL recognition accuracy. The system achieves over 90% accuracy in ASL recognition under controlled conditions, performing well across different environments. The robot-hand learns signs by observing users perform them and can replicate each sign consistently after seeing it once. Future improvements aim to expand the system's vocabulary to include more gestures and enhance its robustness in diverse real-world settings. HandTalk offers a promising solution for automatic two-way ASL translation, with the potential to improve communication for millions of deaf individuals worldwide.
Awards Won: