쓰기 평가에서 채점 특성에 대한 피드백이 채점에 미치는 영향 연구 = The Effects of Individualized Feedback on Rating Behaviors in Korean Writing Assessment

다국어 초록 (Multilingual Abstract)

Kang Min-suk & Sim Je-kyoung. 2015. 12. 31. The Effects of Individualized Feedback on Rating Behaviors in Korean Writing Assessment. Bilingual Research 61, 1-29. The purpose of this study is to investigate the effect of individualized feedback to raters of Korean writing assessment on their severity, consistency, and bias. 20 scripts were rated independently by 22 Korean language teachers. The many-facet Rasch model was used to generate individualized feedback reports on each rater’s relative severity, overall consistency, and significant bias patterns with respect to particular categories of the rating scale. Reports were given and explicated to each rater at a feedback session. Raters were then asked to rate a further set of 20 scripts. A comparison of ratings before and after feedback revealed that individual feedback had positive effects on their ratings (again using the many-facet Rasch model). The range of raters’ severity was reduced after the feedback session, indicating a higher level of agreement between raters compared to the ratings before the feedback. The individualized feedback also had an influence on the rater consistency; raters who were classified as over-fitted or mis-fitted improved their consistency after receiving the feedback. In addition, rater bias in relation to particular categories was all removed. (Korea University)

참고문헌 (Reference)

1 이인혜, "한국어 쓰기 평가의 채점 방식에 따른 채점자 신뢰도 연구 : 종합적 채점 및 분석적 채점을 중심으로" 고려대학교 대학원 2012

2 허유리, "한국어 쓰기 평가에서 채점자의 신뢰도 향상을 위한 방안 연구" 동국대학교 대학원 2014

3 김정숙, "한국어 쓰기 능력 평가 방안- 종합적 채점과 분석적 채점 결과를 중심으로" 이중언어학회 (43) : 81-99, 2010

4 최윤곤, "한국어 교사의 채점자 훈련 모형 개발" 국제한국언어문화학회 10 (10): 295-316, 2013

5 최은지, "한국어 교사의 작문 평가 과정에 대한 사고 구술 연구" 우리어문학회 (47) : 273-300, 2013

6 이인혜, "한국어 교사의 쓰기 평가 효능감과 평가 특성 연구" 이중언어학회 (56) : 231-266, 2014

7 최연희, "채점자 훈련이 영어 작문 채점에 미치는 효과에 대한 연구: FACETS 분석을 통한 신뢰도 변화 분석" 한국응용언어학회 18 (18): 13-292, 2002

8 박영민, "예비국어교사의 중학생 논설문 평가에서 발견되는 엄격성 및 일관성의 특성" 국어교육학회 (43) : 253-283, 2012

9 장소영, "언어교육평가 연구를 위한 FACETS 프로그램." 글로벌콘텐츠 2009

10 이정화, "쓰기 採點의 信賴度 確保 方案 －고급 한국어 학습자 主觀式 쓰기 문항 採點 實驗을 바탕으로－" 한국어문교육연구회 38 (38): 497-522, 2010

11 김동수, "사고 구술법을 통한 한국어 쓰기 평가자의 숙련도별 평가 과정 연구" 고려대학교 대학원 2014

12 강민석, "다국면 라쉬 모형을 통한 한국어 쓰기 평가에서의 채점 경향 연구" 고려대학교 2015

13 Tyndall, B., "Validation in language testing" Multilingual Matters 1996

14 Weigle, S. C., "Using FACETS to model rater training effects" 15 (15): 263-287, 1998

15 Shohamy, E, "The effect of raters’ background and training on the reliability of direct writing tests" 76 : 27-33, 1992

16 Wright, B. D., "Reasonable mean-square fit values" 8 (8): 369-386, 1994

17 Eckes, T., "Rater types in writing performance assessments : A classification approach to rater variability" 25 (25): 155-185, 2008

18 Lumley, T., "Rater characteristics and rater bias : implications for training" 12 (12): 54-71, 1995

19 Schaefer, E., "Rater bias patterns in an EFL writing assessment" 25 (25): 465-493, 2008

20 McNamara, T., "Measuring second language performance" Longman 1996

21 Linacre, J. M., "Many-Facet Rasch Measurement" Mesa Press 1994

22 McNamara, T., "Item Response Theory and the validation of an ESP test for health professionals" 7 (7): 52-76, 1990

23 Eckes, T., "Introduction to Many-Facet Rasch Measurement : analyzing and evaluating rater-mediated assessments" Peter Lang 2011

24 Elder, C., "Individual Feedback to Enhance Rater Training : Does It Work?" 2 (2): 175-196, 2005

25 Linacre, J. M., "Facets computer program for many-facet Rasch measurement" Winsteps. com 2014

26 Ebel, R. L., "Essentials of educational measurement" Prentice Hall 1991

27 Hamp-Lyons, L., "Communicative writing profiles : An investigation of the transferability of a multiple-trait scoring instrument across ESL writing assessment contexts" 41 (41): 337-373, 1991