Booth Id:
TECA019
Category:
Technology Enhances the Arts
Year:
2025
Finalist Names:
Sen, Ishaan (School: Edgewood Junior Senior High School)
Abstract:
Over 2.1 million people in the United States live with limb loss, with approximately 185,000 amputations occurring annually. Limb differences often pose challenges in using input devices such as keyboards. While prosthetic technology has advanced, many low-cost myoelectric prosthetics lack fine motor control, restricting precise arm-based communication.
This project presents a custom-built CNN-Transformer hybrid model designed to recognize American Sign Language (ASL) finger-spelling gestures from five forearm muscle signals. A low-cost surface electromyography (sEMG) apparatus was developed to capture these signals with higher accuracy than comparable commercial systems. The model translates ASL finger gestures into keyboard inputs, providing an alternative digital communication method.
The architecture employs five independent CNN modules—one per sEMG channel—to extract localized signal features, which are then processed by a shared transformer. This design integrates information across all channels, capturing temporal dependencies and broader gesture-related patterns while improving efficiency and reducing data requirements.
A dataset of 10,400 samples was collected from two human test subjects using a custom graphical user interface (GUI) that synchronized data acquisition with labeled ASL gestures. The dataset underwent automated labeling, augmentation, and a train-test split for neural network training.
The system's performance was evaluated using classification accuracy and confusion matrices. Analysis of errors revealed that misclassifications primarily occurred between highly similar gestures. Despite these challenges, the model achieved an accuracy exceeding 85% when tested on hundreds of independent samples.
Awards Won: