Languages are best learned in immersive environments with rich feedback. This is specially true for signed languages due to their visual and poly-componential nature. Computer Aided Language Learning (CALL) solutions successfully incorporate feedback for spoken languages, but no such solution exists for signed languages. Current Sign Language Recognition (SLR) systems are not interpretable and hence not applicable to provide effective feedback to learners. In this work, we propose a modular and explainable machine learning system that is able to provide fine-grained and effective feedback on location, movement and hand-shape to learners of American Sign Language. In addition, we also propose a waterfall architecture for combining the sub-modules to prevent cognitive overload for learners and to decrease time required to provide feedback. The system has an overall accuracy of 87.9 % on real-world data consisting of 25 signs with 3 repetitions each collected from 100 learners.
Here is a version(still being updated) of the paper being presented at IUI 2019. Final version will be uploaded after the conference.
Initially, to gather user opinion we conducted a survey of 52 new learners of American Sign Language at a University. The results of the survey is summarized in the Figure below. There were 29 males and 21 females within the age group of 18-40. The survey was conducted around August 2018.
