정보 탐색 방식에 따른 사용자 경험 비교: 키워드, 규칙 기반, LLM 기반 챗봇을 중심으로 = Exploring User Experiences Across Information Retrieval Interfaces: Keyword Search, Rule-Based, and LLM-Based Chatbots

다국어 초록 (Multilingual Abstract)

Objective: This study aims to evaluate and compare the user experience of three search modalities: keyword-based search, rule-based chatbots, and large language model (LLM)-based chatbots, especially in the context of healthcare information retrieval. The goal is to identify the strengths, weaknesses, and potential opportunities to improve the user experience of each modality.
Background: With the rapid growth of digital transformation, efficient search technologies have become essential. Different search modalities offer unique advantages and constraints. Traditional keyword-based searches are most widely used but limited in handling complex or domain-specific queries. In contrast, rule-based chatbots offer structured, guided interactions, while LLM-based chatbots enable flexible, conversational interactions through advanced natural language processing.
In specialized fields such as healthcare, users often struggle with information overload and a lack of specialized knowledge, highlighting the need for more intuitive retrieval systems.
Method: A total of 60 participants were asked to complete a series of cancer-related information retrieval tasks using three distinct search modalities. The experiment measured task completion time, error rates, workload and user-perceived usability.
Subjective assessments were conducted using the NASA-TLX and Likert-scale questionnaires. Statistical comparisons among modalities were performed using ANOVA.
Results: The ANOVA results revealed significant differences among the three modalities. Notably, rule-based chatbots demonstrated the shortest task completion times and the lowest error rates, whereas LLM-based chatbots suffered from information overload that resulted in higher error rates. Keyword-based search required greater cognitive effort due to manual query input and navigation complexities.
Conclusion: Rule-based chatbots effectively reduce cognitive load and errors in structured search environments, while LLM-based chatbots show promise in handling complex queries but require improvements to mitigate information overload. Although keyword-based search systems remain useful for simple queries, they are less efficient for specialized searches. The findings suggest that developing hybrid chatbots with integrating the structured strengths of rule-based approaches with the flexible natural language capabilities of LLM could optimize information retrieval in healthcare.
Application: The outcomes of this study contribute to designing enhanced information retrieval systems that cater to user needs, particularly in healthcare settings where accuracy and usability are critical. Future research should focus on hybrid chatbot systems that combine the advantages of structured and flexible search modalities.