프롬프트 엔지니어링(Prompt Engineering)을 활용한‘진료수행시험 연습용 챗봇(CPX Practicing Chatbot)’ 시범 개발 = Pilot Development of a 'Clinical Performance Examination (CPX) Practicing Chatbot' Utilizing Prompt Engineering

다국어 초록 (Multilingual Abstract)

Objectives: In the context of competency-based education emphasized in Korean Medicine, this study aimed to develop a pilot version of a CPX (Clinical Performance Examination) Practicing Chatbot utilizing large language models with prompt engineering.
Methods: A standardized patient scenario was acquired from the National Institute of Korean Medicine and transformed into text format. Prompt engineering was then conducted using role prompting and few-shot prompting techniques. The GPT-4 API was employed, and a web application was created using the gradio package. An internal evaluation criterion was established for the quantitative assessment of the chatbot's performance.
Results: The chatbot was implemented and evaluated based on the internal evaluation criterion. It demonstrated relatively high correctness and compliance. However, there is a need for improvement in confidentiality and naturalness.
Conclusions: This study successfully piloted the CPX Practicing Chatbot, revealing the potential for developing educational models using AI technology in the field of Korean Medicine. Additionally, it identified limitations and provided insights for future developmental directions.

참고문헌 (Reference)

1 Chen B, "Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review"

2 곽도원 ; 김민기 ; 권지수 ; 인창식, "Trends in Korean Medical education research from 2003 to 2022" 36 : 89-98, 2023

3 조학준 ; 민성호, "The current status and future operations of Clinical Performance Evaluation(CPX)in the nationwide colleges(graduate schools)of Traditional Korean Medicine" 33 : 9-21, 2020

4 Renze M, "The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Models"

5 Institute of Medical Education and Training, "Revised Clinical Diagnosis" Seoul National University Press 2015

6 Jang D, "GPT-4 can pass the Korean National Licensing Examination for Korean Medicine Doctors" 2 : e0000416-, 2023

7 박사윤 ; 김창업, "Enhancing Korean Medicine Education with Large Language Models : Focusing on the Development of Educational Artificial Intelligence" 37 : 134-138, 2023

8 Cho E, "Developing a manual for clinical practice on real patients in Korean Medicine" 1 : 15-22, 2023

9 Han SY, "Developing a best practice framework for clinical competency education in the traditional East-Asian medicine curriculum" 22 : 2022

10 Banerjee D, "Benchmarking LLM powered Chatbots: Methods and Metrics"

11 한창이 ; 강동원 ; 박중군 ; 김봉현 ; 김규석 ; 김윤범 ; 남혜정, "An Analysis of Clerkship Satisfaction in College of Korean Medicine : Focusing on Doctor-patient Role-play and mock CPX" 33 : 12-24, 2020

12 조학준 ; 노정두 ; 성현경 ; 박정수, "A Survey on Students' Perception of Clinical Performance Examination(CPX)in College of Korean Medicine Using Student Standardized Patients" 24 : 1-13, 2020

13 정선형 ; 김정필 ; 강유정 ; 정혜인 ; 김경한, "A Survey of Recognitions and Satisfaction with Education in Traditional Korean Medicine" 24 : 49-56, 2020

14 Bae H, "A Novel Framework for Understanding the Pattern Identification of Traditional Asian Medicine From the Machine Learning Perspective" 8 : 763533-, 2022

15 임철일 ; 한형종 ; 홍지성 ; 강연석, "2016 Competency Modeling for Doctor of Korean Medicine & Application Plans" 37 : 101-113, 2016