임상수행능력평가(CPX)에서 학교효과가 신뢰도에 미치는 영향 = The School Effect on the Reliability of Clinical Performance Examination in Medical Schools

다국어 초록 (Multilingual Abstract)

Purpose: The purpose of this study is to test the reliability of the clinical performance examination (CPX) using Generalizability theory (G-theory). Through G-theory, the effects of not only students and tasks but also the school will be analyzed as primary sources of error, which can affect the interpretation of the reliability of the CPX.
Methods: One thousand three hundred nineteen students from 16 medical schools that participated in the Seoul-Gyeonggi CPX Consortium 2008 were enrolled. In our research design, we suppose that student is nested within school and crossed with task.
Data analysis was conducted with urGenova.
Results: According to our analysis, the percentage of error variance was 6.2% for school, 14.9% for student nested within school,14.4% for task, and 3% for interaction between school and task. An effect of school on students was observed, but the interaction between task and school was insignificant. When student is nested within school, the universe score decreased and the g-coefficient was less than the g-coefficient of the p × t (p: studentm, t: task) design.
Conclusion: The results show that generalizability theory is useful in detecting various error components in the CPX. Using the generalizability theory to improve the technical quality of performance assessments provides us with greater information compared with traditional test theories.

참고문헌 (Reference)

1 강애남, "학생들의 동료평가를 활용한 수행평가 결과의 일반화가능도 분석" 한국교육평가학회 19 (19): 107-121, 2006

2 김 선, "진료수행시험 평가자로서 표준화 환자 활용의 타당성" 한국의학교육학회 17 (17): 163-172, 2005

3 임형, "임상수행능력평가(CPX)의 오차요인 탐색과 신뢰도 연구" 한국교육평가학회 18 (18): 27-46, 2005

4 노혜린, "의료윤리 능력 평가로서의 객관구조화 구술시험 개발 경험" 한국의학교육학회 21 (21): 23-33, 2009

5 Cronbach LJ, "The generalizability of behavioral measurement: the theory of generalizability for scores and profiles" John Wiley 1972

6 Brennan RL, "The conventional wisdom about group mean scores" 32 : 385-396, 1995

7 "Seoul-Gyeonggi CPX Consortium"

8 Kim JH, "Role and developmental direction of Seoul-Gyeonggi CPX Consortium" 2007

9 Kang SJ, "Regression analysis" Kyoyookbook 2003

10 Brennan RL, "Manual for urGENOVA version 2.1" Iowa Testing Programs, University of Iowa 2001

11 Crick JE, "Manual for GENOVA: a gerneralized analysis of variation system" Research and Development Division, American College Testing Program 1983

12 Lee YM, "Investigation of error factor of CPX using generalizability theory" Seoul National University Hospital 2005

13 Brennan RL, "Generalizability theory: statistics for social science and public policy" Springer 2001

14 Kim SS, "Generalizability theory" Kyoyookbook 2001

15 Korean Society for Education Evaluation, "Educational evaluation thesaurus" Hakjisa 2004

16 Seong TJ, "Educational evaluation" Hakjisa 2002

17 Nam MH, "An application of the generalizability theory to performance assessment" 9 : 73-93, 1996

18 Park JH, "A psychometric evaluation of CPX in relation to validity and reliability" Ewha Womans University 2008

19 Kim CH, "A generalizability theory approach to investigating rater effects with a national exam composed of constructed-response items" 11 : 141-159, 2008