한국어 말하기 평가에서 고정 오류 연구 = A study of the constant errors in criterion- referenced examinations of Korean speaking.

다국어 초록 (Multilingual Abstract)

Raters’ constant errors, represented mostly as leniency errors and halo effect, have been examined for the L2 Korean speaking assessment by native Korean speakers. Fifteen Korean raters assessed twenty-seven native English, Japanese, and Chinese learners of Korean with three levels of beginning, intermediate, and advanced groups. FACETS program with multi-facets Rasch model has been used to analyze the data. The results are as follows: (1) test-takers’ mother tongue has little impact on Korean raters’ rating patterns in a whole, but they tend to grade leniently to English learners of Korean, (2) test-takers’ levelling by L2 Korean proficiency show statically significant influence on raters’ ratings. The study finds that Korean raters could have meaningfully constant errors on their assessment, even though they don’t agree with committing constant errors in the survey. In details, they have more leniency grading to English learners of Korean, while they have halo effects to the beginning stages of L2 Korean learners. The result implies that Korean raters should carry out the authentic speaking assessment training before the real tests. (Konkuk University)

참고문헌 (Reference)

1 신동일, "후광효과의 이해를 통한 채점 오류 분석 연구" 외국어교육학회 9 (9): 215-232, 2002

2 신동일, "한국의 영어 평가학" 한국문화사 2006

3 김명소, "평가형식 및 척도화 방법이 다면평가에서 관대화 오류 및 후광효과에 미치는 영향" 한국산업및조직심리학회 21 (21): 201-224, 2008

4 강석한, "외국인 한국어 말하기 시험의 평가자 요소가 채점에 미치는 영향" 이중언어학회 (55) : 1-29, 2014

5 신상근, "외국어 평가의 이론과 실제" 한국문화사 2010

6 이호, "영작문에서의 대면평가와 익명평가의 비교 연구" 한국영어학회 9 (9): 403-427, 2009

7 장소영, "언어교육평가 연구를 위한 FACETS 프로그램." 글로벌콘텐츠 2009

8 이호, "귀납적 이분법 척도를 활용한 예비 영어교사의 영어말하기 평가 신뢰도와 효용성에 관한 연구" 한국교원교육학회 27 (27): 215-237, 2010

9 Weigle, S. C., "Using FACETS to model rater training effects" 15 : 263-287, 1998

10 Lee, J., "The effect of task types and task difficulty on Korean college students’ speaking test performance" 31-32, 2014

11 Bretz, D., "The current state of performance appraisal research and practice: Concerns, directions and implications" 18 (18): 321-322, 1992

12 Jones, L., "Self-assessment in a selection situation: An evaluation of different measurement approaches" 75 : 145-161, 2002

13 North, B., "Scaling descriptors for language proficiency scales" 15 (15): 217-262, 1998

14 Murphy, R., "Rater errors and rating accuracy" 74 (74): 619-624, 1989

15 Lumley, T., "Rater characteristics and rater bias: Implication for training" 12 : 54-71, 1995

16 최숙기, "Rasch 모형을 활용한 요약문 평가 준거 개발 및 타당도 분석 －고등학생 설명문 텍스트 요약하기를 중심으로－" 한국독서학회 (25) : 415-451, 2011

17 Bardo, J., "Preliminary assessment of format-specific central tendency and leniency error insummated rating scales" 54 : 227-234, 1982

18 Kim, J-T., "Overview of criterion-referenced test" 251-254, 2015

19 McNamara, T., "Measuring second language performance" Longman 1996

20 Kunnan, A., "Large-scale language assessments" Springer Science 2008

21 Bachman, F., "Language testing in practice: Designing and developing useful language tests" OUP 1996

22 Bernardin, J., "Effects of rater training and diary-keeping on psychometric error in ratings" 63 (63): 301-308, 1978

23 Bechger, T., "Detecting halo effects in performance-based examination" 34 (34): 607-619, 2010

24 Shrock, A., "Criterion-referenced test development" International society for performance improvement 2000

25 Luoma, S., "Assessing speaking" Cambridge University Press 2004

26 Lincare, M., "A user’s guide facets: Rasch-model computer program"

27 Bernardin, J., "A recomparison of behavioral expectation scales to summated scales" 61 : 564-570, 1976

28 Fisicaro, A., "A re-examination of the relation between halo error and accuracy" 73 : 239-244, 1988

29 Pulakos, E., "A comparison of rater training programs: Error training and accuracy training" 69 (69): 581-588, 1984