Agreement rates between automated essay scoring systems and human raters: Meta-analysis = 에세이 자동 채점 프로그램과 사람 채점자 간 일치도에 관한 메타분석

다국어 초록 (Multilingual Abstract)

Automated essay scoring (AES) is defined as the scoring of written prose using computer technology. The objective of this meta-analysis is to consider the claim that machine scoring of writing test responses agrees with human raters as much as humans agree with other humans. The effect size is the agreement rate between AES and human scoring estimated using a random effects model. The exact agreement rate between AES and human scoring is 52%, compared with an exact agreement rate of 54% between humans. The adjacent agreement rate between AES and human scoring is 93%, compared to an adjacent agreement rate of 94% between humans. This meta-analysis shows that the agreement rate between AES and human raters is very comparable. This study also compares the subgroup analysis of agreement rates using study characteristic variables such as publication status, AES type, essay type, exam type, human expertise, country, and school level. Implications of the results and potential future research are discussed in the conclusion.

국문 초록 (Abstract)

에세이 자동 채점은 컴퓨터 기술을 이용한 작문 채점이다. 사람 채점과 자동 채점 간의 일치도에 대한 타당화 연구가 많이 진행중이다. 아직까지 사람들 간의 채점의 일치도에 비해 부족하다는 비판도 있지만, 많은 연구들은 사람들 간의 채점만큼의 타당성을 주장하고 있다. 이 메타분석의 목표는 에세이 답안 채점에서 사람 채점자 간 일치도와 기계 채점과 사람 채점 간의 일치도를 비교해 보는 것이다.
이 메타분석에서 효과크기는 자동 채점 프로그램과 사람 채점자와의 일치율을 랜덤효과모형을 이용하여 추정하였다. 사람들 간 54%의 완전일치도와 비교하여 자동 채점 프로그램과 사람 채점자 간 완전일치도는 52%였다. 사람들 간 94%의 근접일치도와 비교하여 자동 채점 프로그램과 사람 채점자 간 근접일치도는 93%이다. 이 메타분석은 자동 채점 프로그램과 사람 채점자 간 일치도가 사람 채점자들 간의 일치도와 비교될 만큼 매우 높다는 것을 보여주었다. 이 연구는 또한 출판 여부, 자동 채점 프로그램 유형, 에세이 유형, 시험 유형, 전문가 채점 여무, 국가별, 학교 수준별과 같은 연구 특징 변수별 유목 간 일치도 차이를 비교하였다. 끝으로 이 연구를 통해 에세이 자동 채점 프로그램의 개발 및 적용의 시사점과 향후 연구 방향, 이 연구의 한계에 대해 논의 및 제시하였다.

목차 (Table of Contents)

• Ⅰ. Introduction
• Ⅱ. Materials and methods
• 1. Literature search and inclusion criteria
• 2. Coding of studies
• 3. Computation of effect sizes
• 4. Combining effect sizes across studies
• Ⅲ. Results
• 1. Description of effects
• 2. Overall analysis
• 3. Subgroup analysis
• 4. Meta-regression by publication year
• Ⅳ. Discussion and Conclusion

참고문헌 (Reference)

1 Mazzeo, J., "The equivalence of scores from automated and conventional educational and psychological tests: A review of the literature" Educational Testing Service 1988

2 Page, E. B., "The computer moves into essay grading : updating the ancient test" 76 : 561-565, 1995

3 Bennett, R. E., "Technology and writing assessment: lessons learned from the US national assessment of educational progress" 2006

4 Hedges, L. V., "Statistical methods for meta-analysis" Academic Press 1985

5 Sirin, S. R., "Socioeconomic status and academic achievement : A meta-analytic review of research" 75 (75): 417-453, 2005

6 Kelly, P. A., "Review of the book automated essay scoring : A cross-disciplinary perspective" 30 (30): 66-68, 2006

7 Cooper, H. M., "Research synthesis and meta-analysis : A step-by-step approach" Sage 2010

8 Hamp-Lyons, L., "On second language writing" Lawrence Erlbaum Associates, Inc 2001

9 Chung, G. K., "Methodological approaches to online scoring of essays" 1997

10 Kim, J., "Meta-analysis of equivalence of computerized and P&P tests on ability measures" 1999

11 Borenstein, M., "Introduction to meta-anaylsis" Wiley 2009

12 Landauer, T. K., "How well can passage meaning be derived without using word order? A comparison of Latent Semantic Analysis and humans" 412-417, 1997

13 Vantage Learning., "How does IntelliMetric score essay responses?" 2003

14 Attali, Y., "Exploring the feedback and revision features of Criterion" 2004

15 Coniam, D., "Experimenting with a computer essay-scoring program based on ESL student writing scripts" 212 (212): 259-279, 2009

16 Shermis, M., "Exit assessments: Evaluating writing ability through automated essay scoring" 2002

17 Nichols, P. D., "Evidence for the interpretation and use of scores from an automated essay scorer" 2004

18 Nichols, P. D., "Evidence for the interpretation and use of scores from an automated essay scorer" Pearson Educational Measurement 2005

19 Raminenia, C., "Evaluation of the e-rater scoring engine for the TOEFL independent and integrated prompts" ETS 2012

20 Raminenia, C., "Evaluation of the e-rater scoring engine for the GRE issue and argument prompts" ETS 2012

21 Mead, A. D., "Equivalence of computerized and paper-and-pencil cognitive ability tests : A meta-analysis" 9 : 287-304, 1993

22 Burstein, J., "Enriching automated essay scoring using discourse marking" 2001

23 Kelly, P. A., "Computerizing scoring of essays for analytical writing" 2001

24 Burstein, J., "Computer analysis of essays" 1998

25 Enright, M. K., "Complementing human judgment of essays written by English language learners with e-rater scoring" 27 : 317-334, 2010

26 Wang, S., "Comparability of computer-based and paper-and-pencil testing in K-12 reading assessment : A meta-analysis of testing mode effects" 68 (68): 5-24, 2008

27 Higgins, J., "Cochrane handbook for systematic review of interventions" 2009

28 McCurry, D., "Can machine scoring deal with broad and open writing tests as well as human readers?" 15 : 118-129, 2010

29 Chodorow, M., "Beyond essay length: Evaluating e-rater’s performance on TOEFL essays" ETS 2004

30 Larkey, L. S., "Automatic essay grading using text categorization techniques" 1998

31 Burstein, J., "Automatedessay scoring for nonnative English speakers" 1999

32 Lonsdale, D., "Automated rating of ESL essays" 2003

33 Page, E. B., "Automated essay scoring: A cross-disciplinary perspective" Lawrence Erlbaum Associates 2003

34 Landauer, T. K., "Automated essay scoring: A cross-disciplinary perspective" Lawrence Erlbaum Associates 87-112, 2003

35 Bereiter, C., "Automated essay scoring: A cross disciplinary approach (vii–ix)" Lawrence Erlbaum Associates 2003

36 Elliot, S., "Automated essay scoring: A cross disciplinary approach" Lawrence Erlbaum Associates 71-86, 2003

37 Attali, Y., "Automated essay scoring with e-raterⓇ V. 2" 4 (4): 1-29, 2006

38 Shermis, M., "Automated essay scoring : A cross disciplinary perspective." Lawrence Erlbaum Associates 2003

39 Vantage Learning., "Assessing the accuracy of IntelliMetric for scoring a district-wide writing assessment" 2003

40 Vantage Learning., "Applying IntelliMetric technology to the scoring of 3rd and 8th grade standardized writing assessments" 2001

41 Card, N. A., "Applied meta-analysis for social science research" The Guilford Press 2011

42 Chen, Y., "An unsupervised automated essay scoring system" 25 (25): 61-67, 2010

43 Rudner, L., "An overview of three approaches to scoring written essays by computer" 2001

44 Dikli, S., "An overview of automated scoring of essays" 5 (5): 1-34, 2006

45 Hutchison, D., "An evaluation of computerized essay marking for national curriculum assessment in the UK for 11-year-olds" 38 (38): 977-989, 2007

46 Vantage Learning., "About IntelliMetric" Vantage Learning 2001

47 Bergstrom, B., "Ability measure equivalence of computer adaptive and pencil and paper tests: A research synthesis" 1992

48 Vantage Learning, "A true score study of IntelliMetric accuracy for holistic and dimensional scoring of college entry-level writing program" 2000

49 Vantage Learning., "A true score study of 11th grade student writing responses using IntelliMetric version 9.0" 2003

50 Vantage Learning., "A study of expert scoring, standard human scoring and IntelliMetric scoring accuracy for statewide eighth grade writing responses" 2002

51 Vantage Learning., "A study of expert scoring and IntelliMetric scoring accuracy for dimensional scoring of grade 11 student writing responses" 2000

52 Yang, Y., "A review of strategies for validating computer-automated scoring" 15 : 391-412, 2002

53 Wang, S., "A meta-analysis of testing mode effects in grade K–12 mathematics tests" 67 : 219-238, 2007