3D Human and Hand Skeletal Features with Continuous Hand Gesture Spotting and Classification = 3D Human and Hand Skeletal Features with Continuous Hand Gesture Spotting and Classification

다국어 초록 (Multilingual Abstract)

Hand gestures are one of the most intuitive and natural ways for communication between human and computer. Recently, the role of hand gesture recognition has become more significant in human-computer interaction applications due to its convenience and naturalness. Hand gestures recognition based method is the topic that is increasingly attracted much research and development.
In this paper, we present a novel approach for continuous dynamic hand gesture recognition. Our approach contains two main modules. Firstly, in the gesture spotting module, the video sequence with continuous gestures are pre-segmented into isolated gestures. Secondly, the gesture classification module classifies the segmented gestures. In the gesture spotting module, the motion of the hand palm and finger movements are fed into a Bidirectional Long Short-Term Memory (Bi-LSTM) network for gesture spotting purpose. In the gesture classification module, three residual 3D Convolution Neural Networks based on ResNet architectures (3D_ResNet) and one Long Short-Term Memory (LSTM) network are combined to efficiently utilize the combination of multiple data channels such as RGB, Optical Flow, Depth and 3D position of key joints.
The promising performance of our approach is obtained by experiments conducted on three public datasets – Chalearn LAP ConGD dataset, 20BN-Jester, and NVIDIA Dynamic Hand gesture Dataset. Our approach achieves mean Jaccard Index of 0.6159, which outperforms the state-of-the-art methods on Chalearn LAP ConGD dataset.

목차 (Table of Contents)

• 1. Introduction 1
• 2. Literature Review . 5
• 2.1 Pose estimation . 5
• 2.1.1 2D/3D human pose estimation 5
• 2.1.2 2D/3D hand pose estimation . 7
• 2.2 Isolated hand gesture recognition . 10
• 2.3 Continuous hand gesture recognition . 13
• 3. Convolution neural network and recurrent neural network . 17
• 3.1 Convolution neural network (CNN) and 3D convolution neural network (3D_CNN) 17
• 3.1.1 Artificial Neural Network (ANN) . 17
• 3.1.2 Convolution neural network (CNN) 18
• 3.1.3 3D convolution neural network (3D_CNN) 19
• 3.2 RNN, LSTM and Bi_LSTM network 19
• 3.2.1 Recurrent Neural Networks (RNN) . 19
• 3.2.2 Long-Short Term Memory (LSTM) 20
• 3.2.3 Bidirectional Long Short Term Memory (Bi-LSTM) 21
• 4. Proposed method 23
• 4.1 Hand gesture spotting . 23
• 4.2 Hand gesture classification . 27
• 5. Experiment and valuation 31
• 5.1 System Environment 31
• 5.2 Dataset 31
• 5.3 Training Processes 32
• 5.3.1 Network training for hand gesture spotting. 32
• 5.3.2 Network training for hand gesture recognition. . 34
• 5.4 Results and analysis . 35
• 5.4.1 Hand gesture spotting 35
• 5.4.2 Hand gesture classification 36
• 5.4.3 Continuous hand gesture spotting-classification . 38
• 6. CONCLUSIONS 40
• References 41