안면부 미동 분석을 통한 사회 감성 인식 및 심장 정보 추출 = Extraction of Cardiac Response and Recognition of Social Emotion from Analyzing Facial Micro-movement

다국어 초록 (Multilingual Abstract)

Emotional communication involves interactions through gestures, language, and facial expressions. Interactions are divided into non-verbal and verbal elements. Non-verbal elements have a high influence on effective emotional communication. Among the non-verbal elements, facial expression occupies the largest proportion. Therefore, emotion recognition and evaluation are proceeding using facial movement. Most emotion recognition studies using facial expressions use cameras. However, the movement generated by facial expressions can express the fake emotion. To do this, researchers studied micro-expression. The micro-expression is a very fast and small expression. These micro-expressions are used to determine the real and fake emotions. Existing facial recognition researches have recognized facial muscle movements as macro-expressions. However, in the case of micro expression, it is recognized as a form containing a macro expression. Therefore, both macro- and micro- should be classified correctly in the face. Thus, the accuracy of emotion recognition using facial movement can be improved.
Therefore, this study focuses on the elimination and application of macro-movement. The basic objective and two application methods are the main goals in this study. The basic objective is to develop a system that removes the macro-movements and extracts the correct micro-movement. (1) The first objective is to based on micro-movement infer social emotion recognition. (2) The second objective is to based on micro-movement infer a cardiac response.
The basic section is to detect and remove macro-movement in facial movement. In the first experiment, it creates a strong facial movement in the face and detects and removes the macro-movement. Strong expression movements were induced by performing Ekman's 6 basic. In the performed facial expression, we identified the origin of the macro movement and removed it. As a result, the strong movement of the macro-movement was removed. In the second experiment, it creates a natural facial movement on the face and detects and removes the macro-movement. Natural movements were induced by speaking. In addition, we observed the time when the macro movement occurred and removed it. As a result, we confirmed that the movements were the same when talking and that the macro-movement was removed. This confirms that the correct micro-movement has been extracted.
Part 1 was social emotion recognition using micro-movement. Based on previous studies, four types of intimacy, empathy, competition-cooperation, and focus were selected for social emotion. In the first intimacy induction experiment was conducted by 30 participants. The intimacy was that the couple (friend) and the non-couple (stranger) performed mutual stimulation. As a result, it showed 78.84% (intimacy: 84.61%, non-intimacy: 73.07%), accuracy for intimacy and non-intimacy. In the second empathy induction experiment, 15 participants proceeded. Empathy proceeded with the couple (friend) seeing and executing the partner stimulus and the monitor stimulus. As a result, it showed 100% (empathy: 100%. non-empathy: 100%) accuracy for empathy and non-empathy. In the third competition-cooperation induction experiment, 52 participants proceeded. The competition-cooperation was conducted by a couple (friend) competing against the other and score competition with the computer. As a result, it showed 95.18% (competition: 92.30%, cooperation: 98.07%) accuracy for competition and cooperation. In the fourth focus induction experiment was conducted by 52 participants. The focus was on the couple (friend) to score the target according to the degree of difficulty. As a result, it showed 94.23% (low-focus: 88.46%, high-focus: 100%) accuracy for low-focus and high-focus.
Part 2 was based on micro-movement inferred a cardiac response. In the expression condition experiment, 14 participants in the comparative experiment between the photoplethysmograph sensor and the camera. Photoplethysmograph and camera were simultaneously measured for 3 minutes for signal comparison. As a result, correlation coefficient 0.783 for non expression and 0.770 for expression. In the natural facial movement condition experiment, 50 participants in the comparative experiment between the electrocardiogram sensor and the camera. Electrocardiogram and camera were simultaneously measured for 5 minutes for signal comparison. As a result, correlation coefficient 0.855 for non natural and 0.832 for natural.

keyword: micro-movement, macro-movement, social emotion, cardiac response, facial movement, noise rejection

목차 (Table of Contents)

• 1. Introduction 1
• 1.1. Facial muscle movements and expression 1
• 1.2. Facial micro-movement and vestibular reflex 5
• 1.3. Emotion recognition 11
• 1.4. Physiological response inference 16
• 1.5. Purpose and necessary 18
• 2. Micro-movement extraction 21
• 2.1. Method 30
• 2.2. Experiment and verification 35
• 2.2.1. Participants 35
• 2.2.2. Stimuli 35
• 2.2.3. Environment and procedures 36
• 2.2.4. Verification 37
• 2.3. Result 38
• 2.3.1. Macro-movement condition 38
• 2.3.2. Natural-movement condition 45
• 2.3.3. Extraction verification 51
• 2.4. Implementation system 57
• 3. Part 1 : Social emotion recognition 58
• 3.1. Intimacy and empathy 60
• 3.1.1. Participants 60
• 3.1.2. Experimental stimuli and measures 60
• 3.1.3. Environment and procedures 62
• 3.1.4. Data processing and analysis 63
• 3.1.5. Result 65
• 3.1.5.1. Intimacy condition 65
• 3.1.5.2. Empathy condition 79
• 3.2. Competition-cooperation and focus 93
• 3.2.1. Participants 93
• 3.2.2. Experimental stimuli and measures 93
• 3.2.3. Environment and procedures 95
• 3.2.4. Data processing and analysis 96
• 3.2.5. Result 97
• 3.2.5.1. Competition-cooperation condition 97
• 3.2.5.2. Focus condition 112
• 4. Part 2 : Cardio-cam 126
• 4.1. Method 127
• 4.2. Experiment and verification 129
• 4.2.1. Participants 129
• 4.2.2. Experimental stimuli and measures 129
• 4.2.3. Environment and procedures 130
• 4.2.4. Data processing and analysis 131
• 4.3. Result 132
• 4.3.1. Macro-movement condition 132
• 4.3.2. Natural-movement condition 133
• 5. General discussion 137
• 6. Conclusion 141
• Reference 142