진료수행시험에서 동일한 사례를 연기한 다수의 표준화 환자 간 채점결과 신뢰도 = Inter-rater Reliability in a Clinical Performance Examination Using Multiple Standardized Patientsfor the Same Case

다국어 초록 (Multilingual Abstract)

Purpose: The “standardization” of standardized patients (SP) is one of the most crucial factors
for a successful clinical performance examination (CPX). This study aimed to examine the
inter-rater reliability among SPs who portrayed the same case during a CPX.
Methods: The context was a CPX conducted under the supervision of CPX Seoul-Gyeonggi
Consortium in K medical school in August 2007. K medical school ran 12 stations consisting
of duplicated sets of 6 cases. In total, thirty SPs participated with 5 SPs acting each of the 6
cases. The SPs evaluated the student’s performances in addition to portraying the cases.
ANCOVA (analysis of covariance) was used to compare scores rated by the different SPs. The
dependent variables were the case scores and the 4 subcomponent (history taking, physical
examination, Clinical courtesy, and Patient-physician interaction) scores for each case; the
independent variable was the SPs; and the covariate was the CPX total score.
Results: The Headache and Cough stations showed an acceptable level of reliability.
Otherwise, Weight Loss and Facial Flushing failed to show consistent scores in all 4
subcomponents. Diarrhea and Lt. hemiparesis showed partial consistency. In terms of the
subcomponents, the physical exam scores were most consistent and the patient-physician
interaction scores were most inconsistent.
Conclusion: This study tested the level of “standardization” of one set of CPX cases with
mixed results. The authors hope that our results will contribute to quality assurance of CPX.

참고문헌 (Reference)

1 Martin, J.A., "Who should rate candidates in an objective structured clinical examination?" 71 (71): 170-175, 1996

2 Nieman, L.Z., "Training and validating the use of geriatric simulated patients" 27 : 154-159, 1988

3 Mazor, K.M., "The relationship between checklist scores on a communication OSCE and analogue patients’ perceptions of communication" 10 (10): 37-51, 2005

4 Wang, W., "The effect of fatigue on the accuracy of standardised patients' checklist recording" 8 : 148-151, 1996

5 Kim, S., "The appropriateness of using standardized patients (SPs) assessment scores in clinical performance examination (CPX)" 17 (17): 163-172, 2005

6 de Champlain, A.F., "Standardised patients' accuracy in recording examinees' behaviours using checklists" 72 (72): S85-S87, 1997

7 Ahn, D., "Standard setting in student assessment by criterion referenced evaluation" 13 (13): 41-45, 2001

8 Tamblyn, R., "Sources of unreliability and bias in standardised patient rating" 3 : 74-85, 1991

9 Solomon, D.J., "Source of measurement error in an ECG examination:implications for performance-based assessments" 9 (9): 283-290, 2004

10 Park, H., "Sharing of information among students and its effect on the scores of clinical performance examination" 17 (17): 185-195, 2005

11 Downing, S., "Reliability: on the reproducibility if assessment data" 38 : 1006-1012, 2004

12 Boulet, J.R., "Quality assurance methods for performance-based assessments" 8 (8): 27-47, 2003

13 Pangaro, L.N., "Performance of “standardized examinees” in a standardizedpatient examination of clinical skills" 72 (72): 1008-1011, 1997

14 Brennan, R., "Performance assessments from the perspective of generalizability theory" 24 (24): 339-353, 2000

15 Sung, T.(, "Nonsulhyung gosawa yecheneuinggye silgigosareul wihan chejumjagan silroedo chujeong" 7 (7): 43-56, 1994

16 Auewarakul, C., "Item analysis to improve reliability for an internal medicine undergraduate OSCE" 10 (10): 105-113, 2005

17 Solomon, D.J., "Global rating of student performance in a standardized patient examination: is the whole more than the sum of the parts?" 5 (5): 131-140, 2000

18 Crossley, J., "Generalisability: a key to unlock professional assessment" 36 (36): 972-978, 2002

19 Chesser, A.M., "Factor analysis can be a useful standard setting tool in a high stakes OSCE assessment" 38 (38): 825-831, 2004

20 Park, W., "Correlation of CPX scores with the scores of the clinical clerkship assessments and written examinations" 17 (17): 297-303, 2005

21 Kwon, I., "Comparison of the evaluation results of faculty with those of standardized patients in a clinical performance examination experience" 17 (17): 173-183, 2005

22 Kim, K., "Comparison of interrater consistence and accuracy of estimation for ability parameters among multiple scoring scales" 14 (14): 327-347, 2001

23 "CPX Seoul-Gyeonggi Consortium"

24 Kim, J., "Analysis of the evaluation for clinical performance examination using standardized patients in one medical school" 16 (16): 51-61, 2004

25 Heine, N., "An analysis of standardized patient checklist errors and their effect on student scores" 37 : 99-104, 2003

26 Im, H., "A study of investigating error sources and reliability for clinical performance examination (CPX)." 18 (18): 27-46, 2005