Abstract Search

ISEF | Projects Database | Finalist Abstract

| Print PDF

Beyond Audio: A Multimodal EMG-Visual Speech System for Reconstructing Voice From Silence

Booth Id:
ENBM056

Category:
Biomedical Engineering

Year:
2026

Finalist Names:
Yang, Eunseok (School: Okgwa High School)

Abstract:
Aphonia—complete loss of voice following stroke or laryngeal surgery—affects millions worldwide yet lacks practical, non-invasive communication solutions. This study was deeply motivated by witnessing a grandmother rendered unable to communicate after a stroke, a condition where the voice is permanently silenced while cognition remains intact. This highlighted the urgent need for accessible communication tools. Traditional clinical electromyography (EMG) equipment costs upwards of $15,000, preventing everyday patient use. To address this, I engineered a custom surface EMG (sEMG) PCB for approximately $285, achieving an over 98% cost reduction. To evaluate the system, I personally recorded a small-scale dataset using my custom hardware. With this data, I analyzed the optimal number of sEMG channels. Results surprisingly appeared as 1 channel accuracy was the highest score, above 2-channel and 3-channel configurations. Character Error Rate (CER): 1-channel (51.05%), 2-channel (55.58%), and 3-channel (55.18%). The design was intentionally capped at three channels to maintain practical, burden-free daily wearability. Regarding the fusion of lip video and EMG signals, experiments on this custom data confirmed that Concatenation achieved the lowest CER (58.11%) compared to cross-attention architectures, proving it the most robust strategy under noisy, limited-data conditions. Finally, I collected a multi-subject dataset from four individuals, including one Aphonia patient, to fine-tune Auto-AVSR, a state-of-the-art visual speech model. This integration reduced the baseline Word Error Rate WER from 19.1% to 9.8%. This study proves that a cost-effective, custom-built EMG device combined with optimal fusion architecture yields a highly practical assistive communication system.

Awards Won:
Fourth Award of $600
King Abdulaziz &amp
his Companions Foundation for Giftedness and Creativity: Full Scholarship from King Fahd University of Petroleum and Minerals(KFUPM) (and a $400 cash prize)
King Abdulaziz &amp
his Companions Foundation for Giftedness and Creativity: NOT TO BE READ -- $400 cash prize for each Full Scholarship from King Fahd University award recipient
Midjourney: Eight awards of $2,500