An Efficient Sign Language Recognition (SLR) System Using Camshift Tracker and Hidden Markov Model (HMM)

Abstract

An efficient Sign Language Recognition (SLR) system could facilitate communication with hearing impaired persons by identifying the sign gestures. Similar to regional spoken languages, different regions have developed their own sign gesture representations (for example, American Sign Language (ASL), German Sign Language (GSL), Indian Sign Language (ISL), etc.). Such variations in the hand shapes and movements add many challenges in the recognition process. The overall SLR process can be divided into a number of modules such as hand and face detection, hand tracking, features extraction and gesture recognition. In this paper, we propose a novel end-to-end SLR system from RGB video-sequences. After detecting the skin color from video frames, Camshift tracker is employed to extract the trajectories of hand motion. Next, Hidden Markov Model (HMM) based sequence classification is used to recognize the gestures. A novel approach identifying double and single hand gestures is also proposed. Furthermore new features, from skin region and hand trajectories, are proposed to improve the gesture classification performance. We tested our system on dataset proposed by American Sign Language Linguistic Research Project (ASLLRP) [25], which consists of isolated signs. The experiment results are encouraging. © 2021, The Author(s), under exclusive licence to Springer Nature Singapore Pte Ltd. part of Springer Nature.