객관 구조화 절차 기술 평가에서 채점자로서의 표준화환자의 신뢰도

국문 초록 (Abstract)

본 연구는 절차기술의 객관구조화 진료시험(Objective Structured Clinical Examination)에서 표준화환자가 평 가자의 역할을 수행할 수 있는지 알아보기 위해 신뢰도를 평가하는데 그 목적이 있다. 시험의 주제는 남성 도뇨관 삽관과 창상드레싱 2가지로 정하고, 2년 이상의 객관구조화 진료시험 채점 경력이 있는 표준화환자와 교수 각 4명을 2명씩 짝을 지워 한 주제 당 표준화환자 그룹과 교수 그룹이 동시에 채점하게 하였다. 표준화환자들에게는 술기의 정의, 방법, 주의점, 후유증에 대한 교육이 이루어졌으며 동영상이 포함된 강의, 교수의 시연 후 표준화환자가 직접 실습해보고 교수로부터 되먹임을 받는 순서로 총 8시간( 주제당 4시간)의 교육이 시행되었다. 8명의 평가자 모두 객 관구조화 진료시험 전날 모여 기존의 동영상자료를 이용한 가상 채점으로 1시간동안 채점 표준화를 이루었다. 채점 표는 체크리스트 14문항과 총괄평가 1문항으로 이루어졌다. 한 학생당, 주제당 5분간의 시험 후 2분간의 평가가 이 루어졌다. 표준화환자와 교수간의 분석은 GENOVA program을 이용하였다. 연구 결과 주제 전체에서 G상수는 0.839, 평가자의 신뢰도는 0.946으로 매우 높았다. 표준화환자그룹과 교수그룹 사이의 평가자간 일치도는 체크리스트에서 0.949, 총괄평가에서 0.908이었다. 따라서 적절한 교육이 선행되어진다면 표준화환자도 절차기술의 객관화진료시험에 신뢰할 만한 평가자로 이용되어질 수 있을 것이다.

다국어 초록 (Multilingual Abstract)

The purpose of this study is to investigate whether standardized patient(SP) can be used as a reliable examiner in Objective Structured Clinical Examination(OSCE). 4 SPs and 4 faculties who have more than 2 years experience of OSCE scoring were selected. For 1 assignment 2 members of faculty and 2 SPs were designated as raters. SPs were educated for assessing 2 technical skills, male Foley catheter insertion and wound dressing, for 8 hours (4 hours / day, each topic). The definition, method, cautions and complications for each of procedural skills were covered in the education. Theoretical lectures, video learning, faculty demonstration and practical training on mannequins were employed. The 8 raters were standardized for an hour with simulated OSCE scoring using previous videos on the day before the OSCE. Each assessment was composed of 14 checklists and 1 global rate. The allotted time for each assignment was 5minutes and for evaluation time 2 minutes per student. The evaluation from the faculty and SPs were compared and analyzed with the GENOVA program. The overall generalizability coefficient (G coefficient) was 0.839 from two cases of OASTS. The reliability of the raters was high, 0.946. The inter-rater agreement between faculty group and SP group was 0.949 for checklist and 0.908 for global rating. Therefore SPs can play a role of raters in OSCE for procedural skills, if they are given the appropriate training.

참고문헌 (Reference)

1 성태제, "타당도와 신뢰도" 학지사 137-163, 2002

2 강승호, "신뢰도(Reliability)" 교육과학사 73-80, 2004

3 Martin JA, "Who should rate candidates in an objective structured clinical examination?" 71 : 170-175, 1996

4 Stillman PL, "The use of practical instructors to evaluate a complete physical examination" 1 : 49-54, 1978

5 Newble DI, "The selection and training of examiners for clinical examinations" 14 (14): 345-349, 1980

6 Barrows HS, "The programmed patient: a technique for appraising student performance in clinical neurology" 39 : 802-805, 1964

7 Stillman PL, "Technical issues: Logistics" 68 : 464-468, 1993

8 Lammers RL, "Teaching and assessing procedural skills using simulation: metrics and methodology" 15 : 1079-1087, 2008

9 Friedlich M, "Structured assessment of minor surgical skills (SAMSS) for clinical clerks" 77 (77): S39-S41, 2002

10 Whelan GP, "Scoring standardized patient examinations: lessons learned from the development and administration of the ECFMG Clinical Skills Assessment (CSAⓇ)" 27 (27): 200-220, 2005

11 Downing SM, "Reliability: on the reproducibility of assessment data" 38 : 1006-1012, 2004

12 Lee B, "Recent world trend in performance-based assessments and application of the standardized patient program in Korean medical education" 12 (12): 377-392, 2000

13 Martin JA, "Objective structured assessment of technical skill (OSATS) for surgical residents" 84 : 273-278, 1997

14 Regehr G, "OSCE performance evaluations made by standardized patients: comparing checklist and global rating scores" 10 (10): S135-S137, 1999

15 Crick JE, "Manual of GENOVA: A generalized analysis of variance system" American College Testing Program 1983

16 Friedman Z, "Experience is not enough: repeated breaches in epidural anesthesia aseptic technique by novice operators despite improved skill" 108 : 914-920, 2008

17 Bullock G, "Evaluating procedural skills competence: inter-rater reliability of expert and non-expert observers" 74 (74): 76-78, 1999

18 Goff BA, "Development of an objective structured assessment of technical skills for obstetric and gynecology residents" 96 : 146-150, 2000

19 Regehr G, "Comparing the psychometric properties of checklists and global scales for assessing performance on an OSCE-format Examination" 73 (73): 993-997, 1998

20 Peggy Wallace, "Coaching Standardized Patients for Use in the Assessment of Clinical Competence" Springer Publishing Company 2007

21 Bould MD, "Assessment of procedural skills in anaesthesia" 103 : 472-483, 2009

22 Harden RM, "Assessment of clinical competence using objective structured examination" 1 : 447-451, 1975

23 Barrows HS, "An overview of the uses of standardized patients for teaching and evaluating clinical skills" 68 (68): 443-451, 1993

24 Heine N, "An analysis of standardized patient checklist errors and their effect on student scores" 37 : 99-104, 2003

25 Humphrey-Murto S, "A comparison of physician examiners and trained assessors in a high-stakes OSCE setting" 80 (80): s59-s62, 2005

26 Morgan PJ, "A comparison of global ratings and checklist scores from an undergraduate assessment using an anesthesia simulator" 76 (76): 1053-1055, 2001