Multimodal Affective State Recognition Using Ear-Centric Biosensors and Advanced Deep Learning Hika Barki

다국어 초록 (Multilingual Abstract)

This study proposes advanced frameworks for affective state recognition by ad- dressing the limitations of conventional visual- and vocal-based methods, which often suffer from subjectivity, user inconvenience, privacy concerns, and susceptibility to envi- ronmental noise. Although many previous studies use scalp EEG or wrist/finger-based PPG, these approaches face challenges such as low wearability, heavy instrumentation, and vulnerability to motion artifacts. In contrast, our research focuses on physiological signal–based emotion recognition using custom-designed ear-wearable PPG and EEG sensors. The ear region offers stable signal quality, reduced motion sensitivity, and high user comfort, making it a practical and unobtrusive location for mental stress and multi-class emotion detection. First, we investigate the feasibility of mental stress detection using a custom- designed wearable in-ear photoplethysmography (PPG) biosensor, which enhances signal quality and user convenience by minimizing motion artifacts. We initially established the system’s effectiveness by transforming 1D PPG signals into 2D time-frequency scalo- grams and classifying them with a Convolutional Neural Network (CNN). To further improve classification performance, we explored more sophisticated approaches by eval- uating various time-frequency representations (CWT, PWVD, and STFT) and employ- ing advanced Vision Transformer (ViT) models. A comprehensive analysis of different signal processing configurations and ViT architectures demonstrated that the ViT-based framework achieved superior accuracy, highlighting the benefits of state-of-the-art deep learning models for analyzing physiological signals. Second, building on the success of the in-ear PPG sensor, we extended its ap- plication to the more complex task of cross-subject, multi-class emotion recognition. To address the computational overhead of 2D signal transformations, we developed a lightweight, end-to-end 1D signal processing framework. This architecture integrates a CNN for local feature extraction with a Bidirectional Long Short-Term Memory (BiL- STM) network to model temporal dependencies. An additive self-attention mechanism was incorporated to enable the model to dynamically focus on the most physiologically relevant segments of the PPG signal. The proposed 1D framework achieved state-of-the- art performance in subject-independent emotion recognition, proving that a streamlined, attention-aware architecture can be both highly effective and computationally efficient for wearable affective computing. Finally, to overcome the inherent limitations of a single physiological modality, we designed a multimodal system for enhanced robustness and accuracy. We developed a custom wearable device that integrates the in-ear PPG sensor with a behind-the- ear Electroencephalography (EEG) sensor, capturing complementary information from both the peripheral and central nervous systems. A comprehensive set of features was extracted from both signals, and the most discriminative features were selected using the ReliefF algorithm and fused. An optimized Extreme Gradient Boosting (XGBoost) classifier, fine-tuned using Bayesian Optimization, was employed for classification. The multimodal approach significantly outperformed single-modality systems, achieving the highest accuracy and confirming that fusing EEG and PPG signals provides a more comprehensive and reliable assessment of affective states. Keywords Affective Computing, Emotion Recognition, Mental Stress Detection, Wearable Sensors, In-Ear Photoplethysmography (PPG), Behind-the-Ear EEG, Deep Learning, Multi- modal Fusion

목차 (Table of Contents)

• List of Figures -ix-
• List of Tables -xi-
• Abbreviations -xii-
• Abstract -xiv-
• 1 Introduction -1-
• 1.1 Background and Motivation 1-
• 1.1.1 Anatomical and Practical Advantages of Ear-Centric Sensing -2-
• 1.2 Ear-Centric Physiological Sensing for Affective Computing 5-
• 1.2.1 Photoplethysmography (PPG) in Ear-Canal Applications . -5-
• 1.2.2 Behind-the-Ear Electroencephalography (EEG) Systems 7-
• 1.2.3 Multimodal Ear-Centric Biosensing Integration 8-
• 1.3 Machine Learning Architectures for Affective Recognition 9-
• 1.3.1 Traditional Feature-Based Machine Learning 9-
• 1.3.2 Deep Learning Approaches 10-
• 1.4 Research Objectives and Scope 11-
• 1.5 Thesis Contribution and Organization 12-
• 2 Mental Stress Detection Using a Wearable In-Ear Plethys-
• mography -16-
• 2.1 Introduction 16-
• 2.2 Materials and Methods 19-
• 2.2.1 Proposed Hardware Architecture 19-
• 2.2.2 Experimental Methodology 22-
• 2.2.2.1 Data Acquisition and Protocol 23-
• 2.2.2.2 PPG Signal Preprocessing 24-
• 2.2.2.3 PPG Signal Transformation 30-
• 2.2.2.4 Proposed CNN for Mental Stress Detection 32-
• 2.2.2.5 Performance Evaluation 35-
• 2.3 Experimental Results 35-
• 2.4 Discussion and Conclusions 38-
• 3 Detection and Classification of Mental Stress Using In-Ear
• Plethysmography and a Vision Transformer -42-
• 3.1 Introduction 42-
• 3.2 Proposed Methodology 46-
• 3.2.1 Proposed hardware architecture 46-
• 3.2.2 Proposed mental stress level classification system 47-
• 3.2.2.1 Experimental Data Collection and Protocol 47-
• 3.2.2.2 Noise Removal and Preprocessing 51-
• 3.2.2.3 Time-Frequency Analysis 52-
• 3.2.2.4 Vision Transformer Model 54-
• 3.2.2.5 K-fold Cross-Validation 58-
• 3.2.2.6 Hyperparameter Configuration 58-
• 3.2.2.7 Performance Evaluation 59-
• 3.3 Experimental Results and Analysis 59-
• 3.3.1 Performance Comparison of Various Time-Frequency Meth-
• ods on ViT Models 60-
• 3.3.2 Time Complexity Analysis and Model Efficiency 65-
• 3.4 Discussion 67-
• 3.4.1 Comparative Analysis of Fine-Tuned ViT-b32 and Other
• Deep Learning Models for Stress Level Classification 69-
• 3.4.2 Comparison with state-of-the-art stress classification methods-70-
• 3.5 Conclusion 73-
• 4 An Attention-Enhanced Convolutional-Bidirectional Recur-
• rent Neural Framework for Cross-Subject Emotion Recogni-
• tion via In-Ear Plethysmography -75-
• 4.1 Introduction 75-
• 4.2 Proposed Methodology 79-
• 4.2.1 Data Acquisition 80-
• 4.2.2 Signal Preprocessing 82-
• 4.2.3 Convolutional Neural Network (CNN) Feature Extraction . -83-
• 4.2.4 Bidirectional Long Short-Term Memory (BiLSTM) 84-
• 4.2.5 Additive Self-Attention Mechanism 88-
• 4.2.6 Hyperparameter Tuning 89-
• 4.3 Experimental Results and Discussions 90-
• 4.3.1 Experimental Setup 90-
• 4.3.2 Evaluation Measures 91-
• 4.3.3 Model Efficiency and Optimizer Influence 92-
• 4.3.4 Performance and Interpretability of Self-Attention 93-
• 4.3.5 Comparison with State-of-the-Art Studies 95-
• 4.3.6 Grad-CAM Interpretation of PPG Features 96-
• 4.3.7 Summary and Future Directions 97-
• 4.4 Conclusion 97-
• 5 Optimized XGBoost for Multimodal Affective State Classifi-
• cation Using In-Ear PPG and Behind-the-Ear EEG Signals -100-
• 5.1 Introduction 100-
• 5.2 Proposed Hardware Architecture 105-
• 5.3 Experimental Methodology 106-
• 5.3.1 Experimental Protocol and Data Acquisition 107-
• 5.3.2 Data Splitting and Validation Protocol 110-
• 5.3.3 Data Preprocessing 110-
• 5.3.4 Feature Extraction 111-
• 5.3.5 Feature selection and Fusion 113-
• 5.3.6 Emotion Classification 113-
• 5.3.6.1 Extreme Gradient Boosting (XGBoost) Model . -113-
• 5.3.6.2 Baseline Classification Models 114-
• 5.3.7 Bayesian Optimization for Hyperparameter Tuning 115-
• 5.3.8 Evaluation metrics 116-
• 5.4 Results and Discussion 116-
• 5.4.1 BTE-EEG Signal Quality Validation: Berger Effect 116-
• 5.4.2 Feature Space Visualization and Cluster Analysis 116-
• 5.4.3 Experimental Results and Classification Analysis 117-
• 5.4.4 Comparison with Baseline Classification Models 124-
• 5.4.5 Comparison with the existing studies 126-
• 5.4.6 Practical Implications and Future Directions 130-
• 5.5 Conclusions 131-
• 6 Conclusion -132-
• 7 Future Works -134-
• Bibliography -135-
• 논문요약 -161-
• Acknowledgement -163-
• Publications Based on the Thesis -165-