영어 말하기 평가의 채점자 신뢰성과 편향성 조사: 구술 인터뷰, 컴퓨터 구술시험, 버슨트의 비교를 통하여 = Examining rater reliability and bias in measuring English speaking performance: Through a comparison of scores on an oral interview, a computerized oral test and a Versant

다국어 초록 (Multilingual Abstract)

The purpose of this study was to investigate inter- and intra- rater reliability in aninterview and a computerized oral test. It was also examined whether ratercharacteristics influenced on their reliability and biases, and finally the scores of bothtests were compared with those of the Versant test using an automated computer ratingsystem. For the study, the data from 21 Korean university students and 18 Korean ornative speakers of English raters with various characteristics were collected. Some ofthe main findings from the study were as follows. First, rater severity was significantlydifferent in each test, but each rater consistently graded on both tests suggesting lowerinter-rater reliability and higher intra-rater reliability. Secondly, rater severity wasimpacted by the rater characteristics such as mother tongue, gender, age, and major.
Lastly, there existed a positive correlation among the scores of the three tests,indicating that the scores of human beings and computers are strongly related.

참고문헌 (Reference)

1 주미진, "컴퓨터 매개 영어 말하기 시험이 수험자 발화에 미치는 영향 조사: 정확성, 유창성, 복잡성을 중심으로" 한국영어학회 11 (11): 677-699, 2011

2 신동일, "채점 경향 분석을 위한 Rasch 측정 모형 적용 연구" 8 (8): 249-272, 2001

3 이영식, "영어작문평가에 대한 채점자 훈련의 원리" 55 (55): 201-217, 2000

4 최인철, "영어 의사소통능력의 모의 구술 면접시험 방식 양상 타당성 검증" 16 (16): 215-246, 2000

5 Center for Applied Linguistics, "oral proficiency interview"

6 Clark, J. L. D., "Validation of a tape-mediated ACTFL/ILR scale-based test of Chinese speaking proficiency" 5 : 187-198, 1988

7 Weigle, S. C., "Using FACETS to model rater training effects" 1994

8 Canale, M, "Theoretical basis of communicative approaches to second language teaching and testing" 1 : 1-47, 1980

9 Charney, D., "The validity of using holistic scoring to evaluate writing : A critical overview" 18 (18): 65-81, 1984

10 Shohamy, E., "The validity of direct versus semi direct oral tests" 7 : 99-123, 1994

11 O'Loughlin, K., "The equivalence of direct and semi-direct speaking tests (studies in language testing 13)" Cambridge University Press 2001

12 Shohamy, E., "The effect of ratings background and training on the reliability of direct writing tests" 76 : 27-33, 1992

13 Stansfield, C. W., "The development and validation of the Portuguese speaking test" 73 : 641-651, 1990

14 주미진, "The attitudes of students’ and teachers’ toward a Computerized Oral Test (COT) and a Face-To-Face Interview (FTFI) in a Korean u" 한국언어과학회 14 (14): 171-193, 2007

15 Center for Applied Linguistics, "Testing/assessment, adult ESL assessment, BEST plus"

16 Fulcher, G., "Testing second language speaking" Pearson Longman 2003

17 Brown, A., "Test development process : Test‐takers’ reactions to a tape‐mediated test of proficiency in spoken Japanese" 10 : 277-303, 1993

18 정행, "SOPI를 활용한 영어 구술능력 평가 개발과 결과 분석" 55 (55): 219-241, 2000

19 박종임, "Rasch 모형을 활용한 국어교사의 채점 일관성 변화 양상 및 원인 분석 -중학생 서사문 채점을 중심으로-" 우리어문학회 39 (39): 301-335, 2011

20 McNamara, T. F., "Measuring second language performance" Longman 1996

21 Linacre, J. M., "Many Facet Rasch measurement" MESA Press 1989

22 "English Speaking Proficiency Testing Academy"

23 "Educational Testing Service"

24 Stansfield, C. W., "Current developments in language testing" Regional Language Center 199-209, 1991

25 Kenyon, D. M, "Comparing examinee attitudes toward computerassisted and other proficiency assessments" 5 : 60-83, 2001

26 Weir, C. J., "Communicative language testing" Prentice Hall 1990

27 류경리, "COPI와 PhonePass의 구술 수행평가 타당도 비교 분석" 고려대학교 교육대학원 2006

28 Wolcott, W., "An overview of writing assessment: Theory, research, and practice" National Council of Teachers of English 1988