Facial Expression Recognition with Local Directional Ternary Pattern and Variable-size Blocks

다국어 초록 (Multilingual Abstract)

In this paper, local directional ternary pattern (LDTP) as a new face descriptor is proposed, for facial expression recognition. Information of emotion-related features (i.e., eyes, eyebrows, upper nose, and mouth) is eciently encoded by LDTP using the directional information and ternary pattern in order to accommodate the robustness of edge patterns in the edge region while overcoming weaknesses of edgebased methods in smooth regions. Our proposal, unlike existing histogram-based face description methods that divide the face into several regions and sample the codes uniformly, uses a two level grid to construct the face descriptor while sampling expression-related information at different scales. For this strategy, we proposed active patterns which are more sensitive to positional information and divide face images into normal regions and sub ones which partition more. For non active patterns which are not sensitive positional information, we describe them in normal regions, and for active ones we describe them in sub regions, which help our description assign more spatial information to emotion-related facial features without sampling error occurring in the existing histogram based description. Additionally, we tested the performance of the proposed method for facial expression recognition by the two different strategies (N-person for person independent test and N-fold cross-validation for person dependent test) on six famous databases: CK+, JAFFE,
MMI, CMU-PIE, GEMEP-FERA and BU-3DFE. We found that the directional information is suitable to describe shapes of emotion-related facial features, which makes LDTP a more discriminable and robust pattern than existing methods for facial expression recognition. And, we observed that the use of ternary pattern makes the proposed LDTP produce more reliable and stable codes than existing edge-based methods since it removes uncertainty of directional pattern generated in smooth region. Moreover, we studied that our novel face description using active pattern and sub regions gives better performance of facial expression recognition for certain conditions. For instance, the combinations of the active pattern (n = 4) and the 1 X 2 or 2 X 2 sub regions show better ability of facial expression recognition than LDTP with existing histogram based description. Moreover, we proposed the variable-sized block representation LDTPVBR to maximize the efficiency of the active pattern increasing spatial information. LDTPVBR also shows very good result.

목차 (Table of Contents)

• Chapter 1 Introduction 1
• Chapter 2 Background Study 6
• 2.1 Overview 6
• 2.2 Local Binary Pattern (LBP) 9
• 2.3 Local Directional Pattern (LDP) 11
• 2.4 Local Directional Number Pattern (LDN) 14
• Chapter 3 Local Directional Ternary Pattern 18
• 3.1 Problems of Existing Edge-based Methods 18
• 3.2 Coding Scheme 21
• 3.3 Discriminability of LDTP 25
• Chapter 4 Face Description for LDTP 28
• 4.1 Problems of Histogram-based Face Description 28
• 4.2 Active Patterns 29
• 4.2.1 Active Patterns for Uniform-grid Blocks 31
• 4.2.2 Active Patterns for Variable-sized Blocks 34
• 4.3 Face Description for LDTP with Uniform-grid Blocks 35
• 4.4 Face Description for LDTP with Variable-sized Blocks 36
• 4.4.1 Facial Features of Variable-sized Blocks 38
• 4.4.2 Settings of Variable-sized Blocks 39
• 4.4.3 Optimal sizes of Variable-sized Blocks 42
• 4.4.4 Variable-sized Block Representation 46
• Chapter 5 Facial Expression Recognition Using LDTP 48
• Chapter 6 Experimental Results 50
• 6.1 Extended Cohn-Kanade Results 51
• 6.1.1 Optimal σ for LDTP 52
• 6.1.2 Optimal LDTP and LDTPnNM Parameters for Uniform-grid Block Representation 53
• 6.1.3 Optimal LDTP and LDTPnNM Parameters for Variable-sized Block Representation 55
• 6.1.4 LDTP vs LDTPnNM 60
• 6.1.5 Comparison results 60
• 6.2 JAFFE Results 64
• 6.3 MMI Results 74
• 6.4 CMU-PIE Results 75
• 6.5 GEMEP-FERA Results 79
• 6.6 BU-3DFE Result 80
• 6.7 Korean DB Results 84
• Chapter 7 Conclusions 88
• Bibliography 90