Booth Id:
TECA015
Category:
Technology Enhances the Arts
Year:
2026
Finalist Names:
Alghazzawi, Jad (School: Dar AlFikr Schools)
Abstract:
Current speech-to-sign language translation systems cannot express the emotions behind the words, making it difficult for deaf and hard-of-hearing individuals to fully grasp the speaker's true intentions, which is a significant communication barrier. The aim of this research was to develop EVA-SL, a novel AI-based system that translates speech into sign language in real-time, while simultaneously visualizing the speaker's emotional state. The system integrates OpenAI's Whisper for speech transcription, GPT-based classification for emotion detection, and Librosa for pitch and tone analysis. Detected emotions and pitch intensity are mapped to a dynamic background color using a Hue-Lightness-Saturation (HLS) model, with color intensity increasing with emotional strength, providing an intuitive visual cue grounded in psychological color-emotion research. The system supports both American and Saudi Sign Languages. To evaluate its effectiveness, 25 participants from the deaf and hard-of-hearing community completed a structured 15-question Likert-scale survey and two real-time interaction tasks involving live speech scenarios. The emotion-to-color feature received a satisfaction score of 4.31 out of 5.0. Furthermore, 88% of participants reported that the adaptive colors helped them understand the emotional intent behind spoken words, and 92% valued the integration of translation and emotion visualization in a single system. This research demonstrates that emotion-adaptive color visualization enables deeper empathy and more complete communication for deaf users. EVA-SL has direct real-world applications in classrooms, hospitals, and public service settings where emotionally accurate communication between hearing and deaf individuals is essential.
Awards Won: