이분식 영어 쓰기 평가 문항 난이도 및 변별도 분석: 고전검사이론과 문항반응이론을 활용하여 = Analyzing Item Difficulty and Discrimination in a Dichotomously Scored Writing Test: Focus on Classical Testing Theorem and Item Response Theory

다국어 초록 (Multilingual Abstract)

This paper explored item difficulty, item discrimination, and item-fit information on the 30 dichotomously-scored attributes of a writing test, using classical testing theorem, Rasch simple logistic model, and 2-parameter logistic model. 251 pre-service English teachers participated in a 50-minute argumentative essay writing test. Their essays were rated by experts who referred to the empirically-based and binary-choice rubric. The findings show that ‘concreteness and appropriateness of the supporting ideas tied to topic sentence’ was the most difficult item. Attributes related to appropriate vocabulary and grammar use usually showed a high discrimination index. Finally, all attributes except for ‘the proper use of tense’ were within in-fit range. This study reminds that further studies which employ fine-grained psychometric analyses on the writing test items are warranted.

참고문헌 (Reference)

1 이호, "영작문에서의 대면평가와 익명평가의 비교 연구" 한국영어학회 9 (9): 403-427, 2009

2 이호, "영어논술평가에서의 일반 척도, 세부과업별 척도, 이분식 척도의 비교 연구" 한국영어학회 11 (11): 601-626, 2011

3 이완기, "영어 평가 방법론" 문진미디어 2007

4 성태제, "문항제작 및 분석의 이론과 실제" 학지사 2010

5 홍효기, "문장의 다양성이 영어논술시험 평가에 미치는 영향 연구" 현대영어교육학회 12 (12): 406-427, 2011

6 이영식, "다국면 Rasch 측정의 Facets 프로그램을 활용한 영어 작문 평가의 원어민 채점 검증" 한국영어어문교육학회 20 (20): 475-496, 2014

7 이호, "귀납적 이분법 척도를 활용한 예비 영어교사의 영어말하기 평가 신뢰도와 효용성에 관한 연구" 한국교원교육학회 27 (27): 215-237, 2010

8 Weigle, S. C., "Using FACETS to model rater training effects" 15 (15): 263-287, 1998

9 Baker, F. B., "The basic of item response theory" Heinemann 1985

10 Fulcher, G., "Testing second language speaking" Pearson Education Limited 2003

11 Upshur, J., "Systematic effects in the rating of second-language speaking ability: Test method and learner discourse" 16 : 82-111, 1999

12 전소연, "Student Raters’ Reflection on a L2 Writing Binary-Choice Scale in the Context of Self- and Peer-Assessment" 한국영어어문교육학회 19 (19): 101-123, 2013

13 Bachman, L. F., "Statistical analyses for language assessment" Cambridge University Press 2004

14 Traub, R. E., "Reliability for the social sciences: Theory and applications" Sage Publications 1994

15 Schaefer, E., "Rater bias patterns in an EFL writing assessment" 25 (25): 465-493, 2008

16 심재우, "On Practical Applications of the Rasch Model Analysis to Language Performance Data: Concepts and Procedures for Running Winsteps" 글로벌영어교육학회 17 (17): 221-238, 2012

17 Condon, W., "Looking beyond judging and ranking: Writing assessment as a generative practice" 14 : 16-33, 2009

18 박묘영, "Exploring the Raters’ Bias on an EFL Writing Assessment Using Multi-faceted Rasch Measurement" 글로벌영어교육학회 17 (17): 175-202, 2012

19 황성삼, "EBB를 이용한 중등학교 말하기 평가척도 개발" 한국영어교육학회 63 (63): 307-331, 2008

20 Knoch, U., "Diagnostic assessment of writing: A comparison of two rating scales" 26 (26): 275-304, 2009

21 Kim, Y. H., "Diagnosing EAP writing ability using the reduced reparameterized unified model" 28 (28): 509-541, 2011

22 Upshur, J., "Constructing rating scales for second language tests" 49 : 3-12, 1995

23 Wilson, M., "Constructing measures: An item response modeling approach" Lawrence Erlbaum Associates 2006

24 Jang, E. E., "Cognitive diagnostic assessment of L2 reading comprehension ability: Validity arguments for fusion model application to LanguEdge assessment" 26 (26): 31-73, 2009

25 Weigle, S. C., "Assessing writing" Cambridge University Press 2002

26 Buck, G., "Application of the rule-space procedure to language testing: examining attributes of a free response listening test" 15 (15): 119-157, 1998

27 송대연, "Applicability of an EBB Scale to English Speaking Performance Assessment in the Korean Secondary School" 팬코리아영어교육학회 21 (21): 111-135, 2009

28 박묘영, "An investigation into pre-service and in-service teachers' judgment of English writing performance" 현대영어교육학회 15 (15): 71-90, 2014

29 Kim, Y. H., "An argument-based validity inquiry into the Empirically-Derived Descriptor-Based Diagnostic (EDD) assessment in ESL academic writing" University of Toronto 2010

30 Wu, M. L., "ACER ConQuest version 2.0: Generalized item response modeling software" ACER Press 2007

31 Kondo-Brown, K., "A FACETS analysis of rater bias in measuring Japanese second language writing performance" 19 (19): 3-31, 2002

32 신유선, "A FACETS Analysis of Rater Characteristics and Rater Bias in Measuring L2 Writing Performance" 한국영어어문교육학회 16 (16): 123-142, 2010