Behind-the-Ear Wearable System for Multimodal Affective Computing

다국어 초록 (Multilingual Abstract)

This thesis presents a novel multimodal emotion recognition framework that integrates behind-the-ear (BTE) photoplethysmography (PPG) sig- nals with facial expression data for accurate and real-time emotional state classification. The research aims to develop a compact, low-cost, and wear- able emotion monitoring system capable of unobtrusive deployment in daily life and healthcare settings. To achieve this, a cross-attention-based multi- branch convolutional neural network (MCNN-CA) is proposed to extract, fuse, and classify features from both physiological and visual modalities. The system employs continuous wavelet transform (CWT) to convert one- dimensional PPG signals into two-dimensional time–frequency scalograms, while facial images are processed through a separate CNN branch. The ex- tracted feature representations are then integrated using a cross-attention fusion mechanism that captures inter-modal dependencies and enhances discriminative emotional cues. Experimental data were collected from fif- teen participants using emotionally evocative video stimuli designed to elicit four emotional states—happy, calm, angry, and sad.The model achieved an overall classification accuracy of 91.31% and an F1-score of 0.919, exceed- ing the performance of multimodal fusion baselines including ResNet50 (87.99%) and DenseNet121 (86.04%). Additionally, the proposed system demonstrated robust subject-independent performance under Leave-One- Subject-Out validation, confirming its generalizability across individuals. The developed wearable device and deep learning framework provide an effective platform for real-time emotion recognition, with potential applica- tions in affective computing, human–computer interaction, telehealth, and mental health monitoring. Future work will focus on extending the model for continuous emotion tracking and integration with additional physiologi- cal signals such as respiration rate and skin conductance for comprehensive emotional assessment.

국문 초록 (Abstract)

본 논문은 귀 뒤(Behind-the-Ear, BTE) 광용적맥파(Photoplethysmography,
PPG) 신호와 얼굴 표정 데이터를 통합하여 정확하명시 실시간 감정 상태를 분류하
는 새로운 다중 모달 감정 인식 프레임워크를 제안한다. 본 연구의 목적은 일상생활
과 헬스케어 환경에서 비침습적으로 사용할 수 있는 소형 · 저비용 웨어러블 감정
모니터링 시스템을 개발하는 것이다. 이를 위해 생리적 신호와 시각적 모달리티로
부터 특징을 추출, 융합 및 분류하기 위한 교차-주의 기반 다중분기 합성곱 신경망 모
델(Multi-branch Convolutional Neural Network with Cross-Attention, MCNN-
CA)을 제안하였다. 시스템은 1차원 PPG 신호를 연속 웨이블릿 변환(Continuous
Wavelet Transform, CWT)을 통해 2차원 시–주파수 스펙트로 그램으로 변환하고,
얼굴 이미지는 별도의 CNN 브랜치를 통해 처리된다. 추출된 특징들은 교차 주의
융합 메커니즘을 통해 상호 모달 의존성을 포착하며 감정 구 분 능력을 향상시킨
다. 실험 데이터는 네 가지 감정 상태(기쁨, 평온, 분노, 슬픔) 를 유도하는 감정
자극 영상을 이용해 15명의 참여자로부터 수집되었으며, 제안 된 모델은 전체 분류
정확도 91.31%와 F1-score 0.919를 달성하여 ResNet50 (87.99%) 및 DenseNet121
(86.04%) 등 동일한 PPG–얼굴 기반 다중모달 융합 모델보다 우수한 성능을 보
였다. 또한 피험자 독립 검증 (Leave-One-Subject-Out Validation)에서도 강인한
일반화 성능을 입증하였다. 개발된 웨어러블 장치와 딥러닝 프레임워크는 실시간
감정 인식에 효과적인 플랫 폼을 제공하며, 감성 컴퓨팅(Affective Computing),
인간–컴퓨터 상호작용(HCI), 원격의료(telehealth), 정신건강 모니터링 등 다양한
응용 가능성을 지닌다. 향후 연구에서는 연속적인 감정 추적 및 호흡률, 피부 전도
도 등의 추가 생리 신호를 통합하여 보다 포괄적인 감정 평가 시스템으로 확장할
예정이다. .

목차 (Table of Contents)

• 1. Introduction 1
• 1.1.Background and Motivation 1
• 1.2.Problem Statement 2
• 1.3.Objectives and Contributions 3
• 2. Literature Review 5
• 2.1.Affective Computing and Emotion Representation 5
• 2.1.1.Models of Emotion 5
• 2.2.Emotion Recognition Using Physiological Signals 7
• 2.3.Emotion Recognition Using Facial Expressions 7
• 2.4.Multimodal Emotion Recognition 8
• 2.5.Research Gaps and Motivation 8
• 3. System Design and Data Acquisition 10
• 3.1.Overview of Proposed System Architecture 10
• 3.2.Behind-the-Ear Wearable Biosensor Design 11
• 3.3.Camera-Based Module for Facial Data Collection 13
• 3.4.Experimental Protocol 15
• 4. Data Preprocessing and Data Alignment 17
• 4.1.Overview 17
• 4.2.PPG Signal Preprocessing 17
• 4.3.Time–Frequency Representation using Continuous Wavelet Trans- form (CWT) 21
• 4.4.Facial Frame Extraction and Alignment 23
• 4.5.Data Alignment and Synchronization 24
• 4.6.Summary 25
• 5. Deep Learning Frameworks 26
• 5.1.Baseline Architectures 26
• 5.1.1.ResNet-50 26
• 5.1.2.DenseNet-121 27
• 5.2.Proposed Model: Multi-Branch Cross-Attention CNN (MCNN-CA) 28
• 5.2.1.Feature Extraction 29
• 5.3.Hyperparameter Optimization and Training Setup 34
• 5.4.Evaluation Metrics 35
• 6. Experimental Results 36
• 6.1.Data Splitting and Validation 37
• 6.2.Performance on Facial Expression Data 37
• 6.2.1.Model Performance on PPG Data 39
• 6.2.2.Emotion Recognition Through Multimodal Feature Fusion (K-Fold) 39
• 6.2.3.Emotion Recognition Through Multimodal Feature Fusion (LOSO) 41
• 6.2.4.Performance of Cross-Attention CNN with Multimodal Fusion 45
• 6.2.5.Model Verification with Preprocessed Inputs 46
• 6.2.6.Time Complexity Analysis and Model Efficiency 47
• 7. Conclusion and Discussion 49
• 7.1.Key Findings and Interpretations 49
• 7.2.Comparison with Previous Studies 49
• 7.3.Advantages of the Behind-the-Ear Multimodal Framework 51
• 7.4.Limitations of the Current Work 52
• 7.5.Potential Applications in Healthcare and HCI 52
• 7.6.Conclusion 53
• Bibliography 54
• 초 록 60
• Acknowledgement 61
• Publications 63