Booth Id:
ROBO027T
Category:
Robotics and Intelligent Machines
Year:
2023
Finalist Names:
Ong, Yu Xi (School: Raffles Institution)
Rajaram, Mirdhini (School: Raffles Institution)
Abstract:
In the new normal, teleconferencing has become increasingly adopted. However, people adopting sign language depend on interpreters in live online interactions, compromising their independence, privacy and flexibility. Existing alternatives such as sign to text and sign to speech automated translators do not embody a crucial component of authentic interactions - emotions. Current emotional speech models do not produce intelligible results with clear emotion. As such, this study investigates the feasibility of a sign language to emotional speech translator to enable signers to be expressive through digital communication technologies. The proposed two-pronged Sign2Speak! model converts sign language to text (Sign2Text) and finally, emotional speech (Text2Speak) based on the emotion identified on the signer’s face.
Using self-recorded datasets for sign language processing (55 individuals) and emotional speech (angry, happy and sad), the model was trained using an LSTM and Tacotron2 architecture. For this project, the vocabulary is limited to 15 conversational words, with more to come in the future. Additional experiments were done to improve the models’ performance. Sign2Text achieved an accuracy of 96% while Text2Speak achieved an accuracy of 100% in terms of clarity against an automated speech transcription, where all generated speech were interpreted correctly, and an accuracy of 80% for emotional speech when tested against a support-vector classifier for emotions. The Sign2Speak! model can be integrated in an app and used in livestreams for natural real-life conversations, online learning, telemedicine and emergency response. In future, a Speak2Sign! model could enable expressive 2-way communications between signers and non-signers.
Awards Won:
Association for the Advancement of Artificial Intelligence: Honorable Mention (do not read aloud). Winners receive a student level membership. Information is included separately in the SAO Portal.
Association for the Advancement of Artificial Intelligence: AAAI Student Memberships for each finalist that is part of the 1st, 2nd, and 3rd Prize Winning projects and 5 Honorable Mention winning projects (up to 3 students per project) (in-kind award / part of the 1st-3rd prize)
Association for the Advancement of Artificial Intelligence: AAAI Student Memberships for each finalist that is part of the 1st, 2nd, and 3rd Prize Winning projects and 5 Honorable Mention winning projects (up to 3 students per project) (in-kind award / part of the 1st-3rd prize)