LLM의 추론 신뢰성 향상을 위한 프롬프트 엔지니어링과 Post-Training 기반의 Safety Alignment 연구

다국어 초록 (Multilingual Abstract)

This study explores how human moral value systems can influence the general reasoning and problem-solving abilities of Large Language Models （LLMs）. While existing research in AI ethics has primarily treated morality as a constraint for ensuring model safety, this thesis extends its role to a cognitive function, empirically analyzing how moral conditioning can alter the model’s reasoning patterns and decision-making processes.
Drawing on Kohlberg's theory of moral development, this study defines three moral conditions aligned with the theory's three stages. Through moral conditioning prompts designed to reflect the core moral values and decision rules of each stage, models are guided to adopt the corresponding patterns of moral reasoning. The validity of this moral alignment is verified using the Defining Issues Test, a psychological assessment tool that measures moral judgment maturity. Models under different moral conditions are then evaluated on the MMLU benchmark.
Experimental results demonstrate that moral perspectives induce meaningful changes in the model’s general reasoning processes. Models conditioned to reflect more advanced moral values exhibit greater self-reflective reasoning and provide more concrete justifications in their responses, which in turn leads to improved MMLU performance. These patterns are consistently observed in additional experiments designed to evaluate the generalizability of moral conditioning effects. Specifically, experiments that vary the surface form of the moral-conditioning prompts, as well as experiments that apply training-based conditioning methods such as Supervised Fine-tuning （SFT） and Direct Preference Optimization （DPO）, show that the model reliably reproduces the intended moral reasoning patterns, with corresponding improvements in MMLU performance. Layer-wise analyses of hidden representations further reveal that higher-level moral conditioning induces larger internal changes during the reasoning process. These internal shifts align with the observed performance gains, suggesting that moral conditioning influences not only the model’s explicit outputs but also its internal representations and information-processing mechanisms.
By examining the cognitive role of moral values, this study demonstrates their potential influence on the general reasoning ability of LLMs. This perspective moves beyond treating morality as merely a safety constraint, revealing it as a factor that can shape the model's reasoning processes.

목차 (Table of Contents)

• 제1장 서론 1
• 1.1. 연구 배경 및 목적 1
• 1.2. 논문의 구성 4
• 제2장 이론적 배경 및 관련 연구 6
• 2.1. 콜버그의 도덕 발달 이론 6
• 2.2. Defining Issues Test (DIT) 7
• 2.3. LLM의 도덕성에 대한 기존 연구 8
• 2.4. LLM Alignment와 행동 제어 기법 9
• 제3장 프롬프트를 통한 도덕성 조정 11
• 3.1. 도덕 조건 정의 11
• 3.2. 도덕적 조건화 프롬프트 설계 12
• 3.3. DIT를 통한 도덕적 정렬 검증 실험 18
• 3.3.1. 실험 설정 18
• 3.3.1.1. 실험 모델 선정 및 환경 18
• 3.3.1.2. 평가 도구 및 딜레마 구성 19
• 3.3.1.3. 평가 방법 23
• 3.3.2. 실험 결과 및 분석 24
• 제4장 도덕적 조건화가 일반 추론 능력에 미치는 영향 27
• 4.1. 실험 설정 27
• 4.1.1. 데이터셋 27
• 4.1.2. 실험 모델 및 평가 방법 28
• 4.2. 정량적 분석: MMLU 정확도 변화 30
• 4.3. 정성적 분석: 추론 패턴의 변화 36
• 제5장 내부 표현 분석 39
• 5.1. 실험 설정 39
• 5.2. 실험 결과 43
• 제6장 추가 실험: 조건화 방법의 확장 46
• 6.1. 프롬프트 변화에 따른 일관성 검증 46
• 6.1.1. 실험 설정 46
• 6.1.2. 실험 결과 47
• 6.2. 조건화 방식에 따른 일관성 검증 48
• 6.2.1. 데이터셋 구성 48
• 6.2.2. 실험 설정 및 결과 51
• 제7장 한계 및 논의 53
• 7.1. 이론적 범위와 제한 53
• 7.2. 일반화 가능성의 한계 53
• 7.3. 해석의 한계 54
• 7.4. 윤리적 논의 54
• 제8장 결론 56
• 참고문헌 57
• ABSTRACT 65