A comprehensive review and ensemble CNN approach for Bharatanatyam single-hand gesture classification

Abstract

Bharatanatyam, the oldest Indian classical dance form, relies on hand gestures to convey meanings and narratives. Recognizing these gestures is important for performance analysis and for supporting novice learners. This study presents the first comprehensive review of classification methods for Bharatanatyam single-hand gestures (Asamyukta Hastas), identifying key limitations in existing work. To address the lack of comprehensive datasets and their limited generalization, this study introduces a robust benchmark dataset collected from a large number of dancers. Previous methods, which relied solely on either deep or hand-crafted features, struggled to capture the fine-grained details of complex gestures. To overcome this, an ensemble Convolutional Neural Network (CNN) based model is proposed to classify 30 Asamyukta Hastas using both deep and handcrafted features. From each raw hand-gesture image, a hand-landmark skeleton image is generated to capture finger positions while removing extraneous details. In addition, an embedded hand-landmark image is produced to provide landmark cues alongside the raw visual features. Three individual CNN models are trained using raw hand gesture images, hand-landmark skeleton images, and embedded hand-landmark images, as each modality provides complementary information. The performance of each CNN is further enhanced using an attention module, and their outputs are ultimately combined through a majority-voting ensemble strategy. The proposed model achieved an average accuracy of 98.28% and an average F1-score of 97.18% on the test set, with a mean recognition time of 325 ms. The source code and dataset for this work are publicly available at https://doi.org/10.5281/zenodo.11514705