Rasch평정척도 경계점을 이용한 기준선 설정 방안 탐색 = Standard setting method using Rasch Rating Scale Model

국문 초록 (Abstract)

본 연구의 목적은 실제 자료를 가지고 전문가에 의한 문항 평정과 학생에 의한 문항 난이도 추정치를 가지고 선형 동등화 방식을 이용해서 피험자의 능력 추정치를 전문가 평정의 분석틀로 동등화 한 다음에 Rasch 평정척도 모형에서 제공되는 척도 추정치에 의해서 기준선을 설정하는 절차를 보여줌과 아울러 경험적 예시 자료를 분석하는 과정에서 나타나는 문제점 진단 및 이를 기초로 향후 이러한 준거기준 설정방법의 교육현장 적용가능성 및 시사점을 도출하는 것이다. 연구 결과, 본 연구에서 제시한 기준선 설정방법을 Angoff 방법 및 Bookmark 방법과 비교해 볼 때 다음의 두 가지 방법적 측면에서 장점을 찾아 볼 수 있었다. 첫째, 반복적인 토의과정을 생략함에 의해서 시간과 평정비용을 줄 일 수 있을 뿐만 아니라, 평정자 개개인의 기준선 설정에 대한 독자적 평정을 보장할 수 있다. 둘째, 기준선을 설정하는 과정마다 통계적인 수치를 제공함에 의해서 기준선 설정에 대한 타당성을 점검할 수 있다. 마지막으로 실제 자료를 가지고 Rasch 평정척도 경계점을 이용해서 기준선을 설정하는 절차를 단계별로 제시하고 해석 하였다.

다국어 초록 (Multilingual Abstract)

The purpose of this study is to introduce standard setting method using Rasch Rating scale model. To do so, the subjects of the study were 141 students in elementary school in Korea and 17 teachers were involved as judges. The procedures to set multiple-cut offs using Rasch Rating scale model in this study were as follows; The material was a reading test comprising 20 items. A group of 17 raters were asked to rate the 20 items, indicating what minimum ability level on the five-point scale a student should exhibit to get each item right. the items were rated 1 to 5, 1 indicating the lowest level and 5 the highest. The items were calibrated using Rasch Rating scale model. The next step was to calibrate the items on the basis of actual student performances. Person measures were converted into the rater frame-of reference by means of the linear equating method. Since the person measures have been brought to the framework of the rater-based analysis thresholds were used to set multiple-cut offs after equating. For its successful implementation in the field of criterion-referenced testing, some suggestions were discussed and implications were given.

목차 (Table of Contents)

• I. 서론
• Ⅱ. 경험적 예시
• Ⅲ. 경험적 예시 분석 결과
• IV. 결론
• 참고문헌
• 초록

참고문헌 (Reference)

1 백순근, "준거지향평가 기준 설정을 위한Rasch 방법의 숙달 학습자 판정 일치도-원점수 및 Angoff 방법과의 비교를 중심으로-" 한국교육평가학회 19 (19): 157-178, 2006

2 Wang,N, "Use of the Rasch IRT model in standard setting: an item-mapping method" Journal of Educational Measurement 40 (40): 231-253, 2003

3 Lewis, D. M, "The bookmark standard setting procedure: Methodology and recent implementation" 1998

4 Mitzel, H. C, "The bookmark method: Psychological perspectives: Setting performance standards: Concepts, methods, and perspectives" Erlbaum 249-281, 2001

5 Impara, J. C, "Teachers' ability to estimate item difficulty: A test of the assumptions in the Angoff standard setting method" Journal of Educational Measurment 35 (35): 69-81, 1998

6 Bejar,I, "Subject matter experts' assessment of item statistics" Applied Psychological Measurement 7 : 303-310, 1983

7 Lewis, D, "Standard setting:A bookmark approach" 1996

8 Engelhard,G, "Standard errors for performance standards based on Bookmark judgments. Rasch measurement: Transactions of the Rasch measurement SIG" American Educational Research Association 21 (21): 1132-1133, 2008

9 Plake,B.S, "Setting performance standards for professional licensure and certification" Applied Measurement in Education 11 (11): 65-80, 1998

10 Angoff,W, "Scales, norms, and equivalent scores: Educational Measurement(2nd ed.)" American Council on Education 508-600, 1971

11 Goodwin,L.D, "Relations between observed item difficulty levels and Angoff minimum passing levels for a group of borderline candidates" Applied Measurement in Education 12 (12): 13-28, 1999

12 Wright, B, "Reasonable mean-square fit values. Rasch measurement: Transactions of the Rasch measurement SIG" American Educational Research Association 8 (8): 370-, 1994

13 Andrich,D, "Rating formulation for ordered response categories" Psychometrika 43 (43): 561-573, 1978

14 Stone,G, "Objective standard setting(or truth in advertising)" Journal of Applied Measurement 2 (2): 187-201, 2001

15 Stone,G, "Objective standard setting" Uinv. of Chicago 1996

16 Wright,B, "How to set standards" Rasch Measurement Transactions 14 (14): 1993

17 Engelhard,G, "Evaluating bookmark judgments. Rasch measurement: Transactions of the Rasch measurement SIG" American Educational Research Association 21 (21): 1097-1098, 2007

18 Mills, C. N, "Estimating and adjusting cutoff scores: Future of selected methods" Applied Measurement in Education 1 : 216-275, 1988

19 Ebel,R.L, "Essentials of educational measurement(2nd ed.)" Prentice-Hall 1972

20 Zieky, M, "Cutscores: A manual for setting standards of performance on educational and occupational tests" ETS 2008

21 Baghaei,P, "Applying the Rasch rating-scale model to set multiple cut-offs. Rasch measurement: Transactions of the Rasch measurement SIG" American Educational Research Association 20 (20): 1075-1076, 2007

22 Jager,R, "An iterative structured judgement process for establishing standards on competency tests: Theory and application" Educational Evaluation and Policy Analysis 4 : 461-476, 1982

23 Nedelsky,L, "Absolute grading standards for objective test" Educational and Psychological Measurement 14 : 3-19, 1954

24 Liancre,J.M, "A user's guide to Winsteps Rasch-model computer programs"