Sentiment-based evaluation of emotional experience in UX : a case study on Duolingo user feedback = UX에서의 감정 경험에 대한 감성 기반 평가 - Duolingo 사용자 피드백 사례 연구 -

다국어 초록 (Multilingual Abstract)

This thesis investigates how emotional user experiences in mobile language-learning applications can be systematically evaluated using large-scale app-store reviews. Focusing on 9,742 Google Play reviews of Duolingo collected between August 2024 and April 2025, the study develops the Sentiment–Emotion–Experience UX (SEE-UX) framework, which integrates sentiment analysis, emotion detection, transformer-based modeling, and feature-level UX mapping. Text preprocessing was followed by a combined lexicon-based sentiment analysis (VADER and TextBlob), emotion extraction using the NRC Emotion Lexicon, and a transformer-based sentiment classifier (DistilBERT) to enhance contextual accuracy. The results reveal a predominantly positive emotional landscape characterized by trust, anticipation, and joy, alongside concentrated negative emotions, particularly anger, sadness, and fear, around specific UX components such as the hearts system, ad interruptions, and subscription prompts. Transformer-based sentiment scores demonstrated stronger alignment with star ratings than lexicon methods, highlighting their value for capturing subtle dissatisfaction masked by polite or mixed-tone language. The study discusses how the SEE-UX framework could be extended to other language-learning applications and outlines cross-app validation as a promising direction for future work. The findings underscore the importance of emotion-aware UX design, offering practical insights for reducing friction, improving motivational features, and designing more empathetic monetization strategies. Overall, the study demonstrates that emotional analytics, when combined with UX theory and modern NLP methods, provide a powerful approach for evaluating and improving digital learning experiences.

국문 초록 (Abstract)

본 연구는 모바일 언어 학습 애플리케이션에서의 사용자 감정 경험을 앱스토어 리
뷰를 기반으로 체계적으로 분석하고 UX 개선으로 연결하는 방법을 탐구한다. 사례 연구 대
상으로 Duolingo의 Google Play 영어 리뷰를 수집하여 전처리한 뒤, VADER와 NRC 감정
어휘 기반 분석과 함께 DistilBERT 기반 문맥적 감성 모델을 적용하였다. 또한 리뷰 내 기
능 관련 표현을 활용해 감정 신호를 구체적 UX 구성요소(예: 하트, 광고, 구독, 스트릭)와
연결하는 SEE-UX(Sentiment–Emotion–Experience for UX) 프레임워크를 제안하였다. 분
석 결과, 전반적으로 긍정 정서가 우세했으나 하트(페널티), 광고 노출, 구독 유도와 같은 기
능은 분노·슬픔·두려움 등 부정 정서를 유발하는 주요 요인으로 나타났다. 또한 변환기 기반
모델은 어휘 기반 접근보다 미묘한 불만과 맥락을 더 잘 포착하여 별점과의 정렬성이 높았
다. 본 연구는 대규모 사용자 피드백에서 감정 기반 UX 인사이트를 도출하고, 제품팀이 실
행 가능한 개선 지침으로 전환할 수 있는 절차적 틀을 제공한다.

목차 (Table of Contents)

• Chapter 1. Introduction 1
• 1.1 Background and Context 1
• 1.2 Problem Statement 2
• 1.3 Research Aim and Objectives 3
• 1.4 Research Questions 4
• 1.5 Significance of the Study 4
• 1.6 Scope and Limitations 5
• 1.7 Analytical Framework: SEE-UX 5
• Chapter 2. Literature Review 7
• 2.1 User Experience and Emotional Quality 7
• 2.2 Emotional Design and Motivation in Digital Products 8
• 2.3 Sentiment and Emotion Analysis of User-Generated Text 9
• 2.4 Transformer-Based Sentiment Analysis Models 9
• 2.5 Mining App-Store Reviews for UX and Design Insights 10
• 2.6 Language-Learning Applications and Duolingo 11
• 2.7 Emerging Need for Emotion-Aware UX Frameworks 11
• 2.8 Conceptual Foundations of the SEE-UX Framework 12
• Chapter 3. Methodology 14
• 3.1 Research Design 14
• 3.2 Data Source and Collection 14
• 3.2.1 Primary Dataset: Duolingo Google Play Reviews 14
• 3.2.2 Metadata Overview 15
• 3.2.3 Inclusion and Screening 15
• 3.3 Data Preprocessing 16
• 3.3.1 Text Normalization 16
• 3.3.2 Tokenization, Stop-Word Removal, and Lemmatization 16
• 3.3.3 Output Formatting and Data Storage 16
• 3.4 Lexicon-Based Sentiment Analysis (VADER + TextBlob) 16
• 3.4.1 Sentiment Tools 17
• 3.4.2 Combined Sentiment Score 17
• 3.4.3 Sentiment Categorization 17
• 3.4.4 Statistical Analysis 17
• 3.5 Transformer-Based Sentiment Analysis 17
• 3.5.1 Motivation and Advantages 18
• 3.5.2 Model Selection 18
• 3.5.3 Implementation Workflow 18
• 3.5.4 Comparative Analysis 19
• 3.6 Emotion Analysis Using the NRC Emotion Lexicon 19
• 3.6.1 Implementation Steps 19
• 3.6.2 Reliability and Validity Checks 19
• 3.7 Mapping Emotional Data to UX Components 19
• 3.7.1 Identifying UX Themes 19
• 3.7.2 Emotional Signature Analysis 20
• 3.8 Statistical and Visual Analysis 20
• 3.9 Integration of Quantitative Analysis with UX Theory 21
• 3.10 Ethical Considerations 21
• 3.11 Summary 21
• Chapter 4. Results 22
• 4.1.1 Rating Distribution 22
• 4.1.2 Temporal Trends in Ratings 23
• 4.2 Lexicon-Based Sentiment Analysis 24
• 4.2.1 Overall Sentiment Distribution 24
• 4.2.2 Sentiment Trends Over Time 25
• 4.3 Relationship Between Sentiment and Ratings 25
• 4.4 Emotion Analysis Using the NRC Emotion Lexicon 26
• 4.4.1 Overall Emotional Profile 26
• 4.4.2 Emotional Differences Across Rating Levels 27
• 4.5 Mapping Emotions to UX Components 28
• 4.5.1 Overview of UX Themes 28
• 4.5.2 Advertisements 29
• 4.5.3 Hearts (Lives System) 29
• 4.5.4 Subscription and Super Duolingo 29
• 4.5.5 Streaks and Gamification 30
• 4.6 Comparative Summary: Lexicon vs. Transformer Models 30
• 4.6.1 Beyond Sentiment: Topic Modeling and Regression Extensions 31
• 4.7 Summary of Key Findings 31
• Chapter 5. Discussion 33
• 5.1 Understanding Duolingo’s Emotional Landscape (RQ1) 33
• 5.2 Emotional Triggers and UX Components (RQ2) 33
• 5.2.1 Hearts and Perceived Punishment 34
• 5.2.2 Advertisements and Emotional Friction 34
• 5.2.3 Subscription and Fairness Perception 34
• 5.2.4 Streaks as Dual-Emotion Drivers 34
• 5.3 Predictive and Explanatory Power of Sentiment and Emotion Metrics (RQ3) 35
• 5.3.1 Lexicon-Based Insights 35
• 5.3.2 Transformative Value of Transformer Models 35
• 5.3.3 Emotions as Diagnostic Indicators 36
• 5.4 Theoretical Implications 36
• 5.4.1 Integrating Emotional Design and Data Science 36
• 5.4.2 Emotions as Indicators of Value and Fairness 36
• 5.4.3 The Dual Nature of Gamified Features 36
• 5.5 Practical Implications for Design and Industry 36
• 5.5.1 Designing Emotionally Supportive Systems 37
• 5.5.2 Implementing Empathetic Monetization 37
• 5.5.3 Real-Time Emotional Monitoring 37
• 5.5.4 Feature-Level Emotional Mapping 37
• 5.5.5 Design Checklist and Implementation Playbook 37
• 5.6 Methodological Reflections 40
• 5.7 Summary 40
• Chapter 6. Conclusion and Future Work 41
• 6.1 Summary of the Study 41
• 6.2 Summary of Key Findings 42
• 6.3 Theoretical Contributions 44
• 6.4 Methodological Contributions 45
• 6.5 Practical Implications 45
• 6.6 Limitations 46
• 6.7 Future Research Directions 47
• 6.8 Final Reflection 48
• References 50
• Korean Abstract 53
• Appendix 54