Booth Id:
TECA018
Category:
Technology Enhances the Arts
Year:
2026
Finalist Names:
Tang, Shiqing (Ellie) (School: Allen High School)
Abstract:
Dementia affects over 57 million individuals worldwide, causing two critical challenges: facial recognition deficit and disordered speech. Current assistive tools for dementia such as facial recognition aids or dementia caregiver apps only offer limited static solutions, which fail to adapt to the patients' changing needs and become less productive as their condition worsens. This project introduces a novel AI agent integrated with augmented reality (AR) glasses that offers facial recognition with on screen prompts, disordered speech correction, and evolves with patients' daily changing data. SAMA is made of three subsystems: (1) a Facial Recognition Self-Evolving System, using the ResNet model to identify known individuals and auto learn/label new faces using Qwen 3 LLM reasoning; (2) an Auto Speech Recognition Self-Evolving System, using the Whisper ASR model to detect disordered speech and continuously label/learn user voice data; and (3) a Multimodal Information Based Second Pass Regeneration System which ranks Whisper's N-best hypotheses using environmental context info to improve prediction accuracy. Furthermore, through multiple iterations of testing and validation, I designed a two stage training pipeline to fine-tune SAMA to auto adapt to daily degraded speech. Stage one utilizes a curriculum learning pipeline to improve Whisper's accuracy in progressively degrading speech, while stage two experimented several training algorithms including SFT and RL training to enhance the text reasoning of Qwen 3 for hypothesis ranking. Testing of the two stage pipeline show significant reduction of WER and facial recognition evaluation also received exemplary accuracy. SAMA presents a unique self-evolving agent to dementia care, benefitting millions of patients world-wide.
Awards Won:
Association for Computing Machinery: Third Award of $1,500