Real-time interpretation of American Sign Language using SSD-MobileNet

Abstract

Individuals with hearing impairments may struggle with integrating into society because the general population does not understand sign language. Consequently, this can lead to isolation and exclusion from social and professional opportunities. To address this issue, this paper proposes a system for the real-time interpretation of American Sign Language (ASL) using computer vision technology. This system uses a normal webcam to detect and interpret 26 letters of the English alphabet and three auxiliary signs. To achieve this goal, the pre-trained lightweight single-shot multibox detection network model, from the TensorFlow object detection application programming interface (API), SSD-MobileNet was used. After the training phase of the proposed model with a personally collected dataset, the obtained results in testing are promising, with a precision of 82.8% and a recall of 85%. The proposed system represents a forward step in sign language translation. Furthermore, it can be adapted to interpret other sign languages.