Dynamic Feature Selection and Voting for Real-time Recognition of Fingerspelled Alphabet using Wearables
- DyFAV: Dynamic Feature Selection and Voting for Real-time Recognition of Fingerspelled Alphabet using Wearables (pdf)
Abstract:
Recent research has shown that reliable recognition of sign language words and phrases using user-friendly and non-invasive armbands is feasible and desirable. This work provides an analysis and implementation of including fingerspelling recognition (FR) in such systems, which is a much harder problem due to lack of distinctive hand movements. A novel algorithm called DyFAV (Dynamic Feature Selection and Voting) is proposed for this purpose that exploits the fact that fingerspelling has a finite corpus (26 letters for ASL). The system uses an independent multiple agent voting approach to identify letters with high accuracy. The independent voting of the agents ensures that the algorithm is highly parallelizable and thus recognition times can be kept low to suit real-time mobile applications. The results are demonstrated on the entire ASL alphabet corpus for nine people with limited training and average recognition accuracy of 95.36% is achieved which is better than the state-of-art for armband sensors. The mobile, non-invasive, and real time nature of the technology is demonstrated by evaluating performance on various types of Android phones and remote server configurations.
Dataset:
Please find below the link to the dataset used for this project. If you use the dataset in your research please refer to the publication:
This work was published on the 2017 ACM IUI . (ACM Digital Library Link, pdf). It received the Best Student Paper, honorable mention.
Dataset Description: (Link)
9 users wore the Myo Armband and data was collected for 5s. for each letter of the alphabet. The first 8 columns contain data for the 8 EMG pods, the next 3 are for Accelerometer, the next 3 are for Gyroscope and the final 3are for Orientation (Roll, Pitch and Yaw)
Why Fingerspelling
Fingerspelling is integrated in sign languages in systematized ways and is very important for overall understanding of sign languages
Other Research Blogs: