도메인 특화 생성형 AI 기반 하이브리드 질의응답 모델의 통합적 성능 분석 연구 = An Integrated Performance Analysis of a Domain-Specific Generative AI?Based Hybrid Q&A Model

다국어 초록 (Multilingual Abstract)

The purpose of this study is to develop and validate a domain-specific question-answering system for product information and technical inquiries based on enterprise internal data, specifically targeting Korean electronics manufacturing and factory automation sectors. As enterprises increasingly adopt artificial intelligence for customer service and technical support, the demand for domain-tailored solutions has surged. Gartner projects that by 2027, over 50% of enterprise generative AI models will be domain-specific, up from 1% in 2023, highlighting the critical need for specialized AI solutions that can handle industry-specific terminology, processes, and knowledge. Existing rule-based chatbots and keyword-based search systems show significant limitations in processing unstructured data and handling complex, context-dependent queries common in technical support scenarios. Meanwhile, generative AI systems based on Large Language Models (LLMs) have the potential to address these challenges through their natural language understanding and generation capabilities; however, concerns regarding data leakage, response reliability, and hallucination continue to pose significant barriers to their adoption in enterprise environments.These issues are particularly pronounced in manufacturing domains where accuracy and trustworthiness are paramount. This study proposes a novel hybrid architecture that strategically combines domain-specific fine-tuning with Advanced Retrieval-Augmented Generation (RAG) to leverage the complementary strengths of both approaches. Fine-tuning using Parameter-Efficient Fine-Tuning (PEFT) techniques internalizes enterprise-specific terminology, product specifications, and technical processes into the model, while RAG provides real-time access to updated documentation and ensures factual grounding. In addition, the proposed architecture integrates an intelligent FAQ matching system with a dynamic routing mechanism, enabling the selection of the optimal response strategy based on the characteristics of each query, thereby achieving both high domain expertise and accurate information delivery. The system was implemented on AWS infrastructure using Claude 3.5 Sonnet for the hybrid architecture and Claude 3 Haiku for the RAG-only configuration. The knowledge base comprises 180 technical documents and 500 question-answer pairs covering product specifications, troubleshooting procedures, and operational guidelines. Evaluation employed a mixed-methods approach combining quantitative performance metrics and qualitative expert assessment through Focus Group Interviews with internal staff and technical specialists. The experimental design compared four distinct configurations: fine-tuning alone, RAG alone, simple parallel implementation, and the proposed hybrid architecture across 118 test questions spanning 12 scenarios with varying query types and formats. Results demonstrate that the hybrid architecture achieves superior performance across all evaluation dimensions. The fine-tuning-only model excelled in domain expertise and terminology accuracy but was limited in retrieving up-to-date information and exhibited a relatively high rate of hallucination. In contrast, the RAG-only model demonstrated strengths in providing current information but lacked deep domain understanding. The hybrid architecture effectively mitigates the weaknesses of each individual approach, achieving balanced and robust performance. Additionally, the intelligent routing mechanism guides queries to the appropriate processing pathways, maximizing response quality while maintaining computational efficiency. This study empirically demonstrates that strategically combining fine-tuning and RAG outperforms individual approaches in domain-specific enterprise applications, thereby making significant theoretical and practical contributions. It addresses critical industry concerns regarding reliability and trustworthiness while providing actionable implementation guidelines for deploying generative AI systems across manufacturing and technical support domains.

목차 (Table of Contents)

• 제 1 장 서론 1
• 제 1 절 연구의 배경 및 목적 1
• 1. 연구의 배경 1
• 2. 연구의 목적 6
• 제 2 절 연구의 범위와 구성 9
• 1. 연구의 범위 9
• 2. 연구의 구성 10
• 제 2 장 이론적 배경 12
• 제 1 절 생성형 AI 12
• 1. 파운데이션 모델 (FM) 17
• 2. 거대 언어 모델 (LLM) 21
• 제 2 절 도메인특화 생성형 AI 23
• 1. 프롬프트 엔지니어링 24
• 2. 파인튜닝(Fine-tuning) 29
• 3. RAG(Retrieval-Augmented Generation) 33
• 4. 하이브리드 아키텍처 38
• 제 3 절 질의 응답시스템 42
• 1. 질의 응답시스템 42
• 2. AI Assistant 44
• 3. 생성형 AI 기반 챗봇 서비스 46
• 4. 도메인 특화 생성형 AI 사례 50
• 제 4 절 생성형 AI 애플리케이션 개발 프레임워크 53
• 1. LangChain 프레임워크 53
• 2. LangGraph 55
• 제 3 장 연구방법 57
• 제 1 절 연구개요 및 프레임워크 57
• 1. 연구개요 57
• 2. 시스템 아키텍처 59
• 3. 하이브리드 RAG 모델 64
• 제 2 절 성능평가 68
• 1. 평가지표 구성 68
• 2. 평가 항목 및 척도 69
• 3. 평가자의 구성 73
• 제 4 장 생성형AI 시스템 개발 74
• 제 1 절 구현개요 및 프레임워크 74
• 1. 구현 개요 74
• 2. 시스템 구축 78
• 제 2 절 데이터 세트 82
• 1. 지식 베이스 데이터 세트 구축 82
• 2. 질의응답 데이터 세트 구축 88
• 제 3 절 생성형 AI 질의 응답 시스템 91
• 1. 사용자 인터페이스 91
• 2. 프롬프트 검증결과 94
• 제 5 장 연구 결과 104
• 제 1 절 평가 개요 및 방법 104
• 제 2 절 시스템 응답 정확도 평가 105
• 제 3 절 하이브리드 아키텍처 성능 비교 107
• 제 6 장 결론 110
• 제 1 절 연구결과의 요약 110
• 제 2 절 연구의 시사점 112
• 제 3 절 연구의 한계점 및 향후 연구 114
• 참고문헌 117
• 부록 Ⅰ. 프롬프트 검증 상세 결과 131
• 부록 Ⅱ. 질의응답 시스템 성능 평가표 135