Embodied Narrative Intelligence: A Novel Approach to Experiential Fiction Generation through Robotic Sensory Fusion and Advanced Language Models = 체화된 서사 지능:로봇 감각 융합과 언어 모델을 통한 경험적 소설 생성에 관한 접근

다국어 초록 (Multilingual Abstract)

Human narratives are formed upon rich sensory experiences encompassing not only vision and hearing but also temperature, wind, and bodily movement. However, existing large language model text generation research has primarily relied on static images or text prompts, failing to adequately reflect the continuous sensor states and embodied experiences that real robots perceive. To address this limitation, this thesis proposes a sensor-conditioned embodied narrative generation framework that enables robots to automat- ically generate first-person literary narratives based on their own sensor states. To this end, we design a sensor encoder that projects a 12-dimensional sensor vector—comprising temperature, humidity, wind direction, 6-axis IMU, and relative angles—into high-dimensional embeddings, and fuse these into the input embedding space of a TinyLLaVA-based multimodal language model. Furthermore, we construct a synthetic sensor-text dataset of 40,000 pairs by combining virtual environments, weather conditions, and robot states, and fine-tune the model to generate first-person literary narratives corresponding to each sensor state. The proposed model receives sensor sequences collected from a real robot as input to gener- ate narratives, and we quantitatively evaluate performance across multiple dimensions—including overall quality, richness of embodied expression, and utilization of sensor states—using three independent LLM evaluators. Experimental results demonstrate that the proposed sensor-conditioned embodied narrative model consistently achieves superior preference ratings and scores compared to the pre-trained baseline model across most evaluation criteria. This research presents a novel form of embodied intelligence in which robots narratively “speak” their own sensory experiences, suggesting diverse future applications such as robot diaries, field experience documentation, and storytelling for human-robot interaction.

목차 (Table of Contents)

• Contents iii
• List of Tables iv
• List of Figures v
• Abstract vi
• 1 Introduction 1
• 1.1 Background and Purpose of the Study 1
• 1.2 Problem Statement and Limitations of Existing LLMs 2
• 1.3 Research Objectives and Contributions 3
• 1.4 Organization of the Thesis 4
• 2 Theoretical Background 5
• 2.1 Large Language Models and Creative Text Generation 5
• 2.2 Vision-Language Models and Multimodal LLMs 6
• 2.3 Embodied AI and Robot Intelligence 8
• 2.4 Non-Visual Sensor-Based Multimodal Learning 9
• 2.5 Literary Theory on Creative Writing and Sensory Expression 11
• 2.6 Synthesis of Prior Research and Distinctiveness of This Study 12
• 3 Problem Definition and Overall Framework 14
• 3.1 Research Objectives and Problem Definition 14
• 3.2 Conceptual Framework for Sensor-Based Embodied Narrative 15
• 3.3 Overview of Sensor-LLM Integration Architecture 16
• 3.4 Research Procedure 18
• 4 Genre Corpus Construction 20
• 4.1 Genre Selection and Corpus Building 20
• 4.2 Genre-Specific Keyword Extraction 21
• 4.3 Sensor Data Sampling Strategy 22
• 4.4 LLM-Based Narrative Generation Pipeline 24
• 4.5 Dual-Prompt Strategy 25
• 5 Sensor-Fused Model Architecture 28
• 5.1 Architecture Overview 28
• 5.2 Sensor Encoder Design 29
• 5.3 Multimodal Embedding Integration 31
• 5.4 Training Strategy and Optimization 32
• 5.5 Inference and Real-Time Application 33
• 6 Robot Platform Implementation 35
• 6.1 Dual Jetson Architecture 35
• 6.2 Robot Hardware Configuration 37
• 6.3 ROS2-Based Control System 38
• 6.4 Sensor Data Collection System 39
• 7 Experimental Design and Evaluation 42
• 7.1 Experimental Objectives and Research Questions 42
• 7.2 Evaluation Dataset Composition 43
• 7.3 LLM-as-a-Judge-Based Automatic Evaluation 44
• 7.4 Evaluation Metric Definitions 45
• 8 Experimental Results and Analysis 47
• 8.1 Overall Performance Comparison 47
• 8.2 Component-wise Performance Analysis 48
• 8.3 Per-Sensor Accuracy Evaluation 50
• 8.4 Inference Efficiency Analysis 51
• 9 Conclusion and Future Work 53
• 9.1 Main Contributions 53
• 9.2 Limitations of the Study 54
• 9.3 Future Research Directions 55
• References 64
• 국문요약 65