Efficacy of Corpus-driven Scoring Methods for Computer-based Dictation Tests

( Inn-chull Choi ),( Seon-yeong Park ) 한국멀티미디어언어교육학회 2016 멀티미디어 언어교육 Vol.19 No.4

The present research is a follow-up research of an exploratory validation study of computer-based dictation tests utilizing TTS (text-to-speech) (Choi & Park, 2015). The study attempted to explore the efficacy of different scoring methods utilizing corpus linguistic techniques. To this end, the data collected from a prior study was employed in the research. 117 Korean EFL learners participated in this research and were divided into three groups depending on educational levels as well as their proficiency levels: 37 UNI (advanced), 40 HSU (intermediate), and 40 HSL (low). Participants` responses were scored using six different methods: GS (graduated scoring), AM (appropriate meaning-based scoring), EW (exact-word scoring), WM (wordlist match scoring), WML (wordlist match scoring including lemmas), and WMS (wordlist match scoring including synonyms). Comparative analysis of the efficacy of the scoring methods revealed that the four methods were highly correlated with each other and that corpus-based wordlist matching was found to be the most effective and robust scoring method. Among the three wordlist match scoring methods producing robust results, WM was proven to be the most effective semi-automated method with high practicality, requiring the least amount of time and energy for scoring.

국가수준 학업성취도 평가 국어 서답형 문항의 자동채점 결과 분석

노은희 ( Eun Hee Noh ) 국어교육학회 2014 國語敎育學硏究 Vol.49 No.2

이 연구는 2013년 개발된 한국어 서답형 문항 자동채점 프로그램 KASS(Korean Automatic Scoring System)을 활용하여 2012년 학업성취도 평가의 초.중.고 국어 총 17문항 각 3,010개 답안을 대상으로 자동채점 결과를 분석하여 향후 국어 서답형 문항의 자동채점 가능성을 탐색하였다. 자동채점 결과, P1~P3에 해당하는 단어ㆍ구 수준 서답형 문항의 Kappa계수는 최소 0.97 이상이며 절반 이상의 문항이 1.00으로 채점 신뢰도가 매우 높게 나타났다. 또한 인간채점과 채점 불일치율도 0~0.5% 사이로 매우 낮게 나타났다. 다만, 요구하는 답안이 길고 복잡해질수록 인간채점과 자동채점 결과의 일치도가 떨어지고 불일치율이 증가한다는 점은 확인할 수 있다. 한국어 처리 기술 및 지식베이스의 여건이 충분하지 않은 상황에서, 자동채점이 용이한 단답형 문항부터 자동채점 시스템을 구축하는 본 연구는 출발점으로서의 의미가 크다. 현재 대규모 평가의 서답형 문항에서 다수를 차지하고 있는 단답형 문항을 우선 처리할 수 있을 뿐 아니라, 이를 바탕으로 좀 더 장기적으로 내용 기반의 서답형 문항에 대한 자동채점 연구도 지속적으로 발전시킬 수 있는 기반을 마련할 수 있기 때문이다. The purpose of this study is to improve the automatic scoring program of Korean supply-type items and to increase application of the program for effectiveness of scoring and reliable scoring. For the trial application of the automatic scoring program, I scored 17 supply-type Korean items of the 2012 NAEA(National Assessments of Educational Achievement). The numbers of answer were 3,010 of each Korean item. The results of the 2012 NAEA items demonstrated that the scoring rate was quite high, 97~100%, and Kappa coefficients were high (at over .97). The rate of scoring errors was 0~0.5%. The error rate of most items was very small (under 1%). The sources of scoring errors were either spelling errors or the non-recognition of analogous terms, and symbols. The ability to use automatic scoring program in operational scoring environments, such as the NAEA, reduces the time and cost associated with having multiple human scorers score answers of supply-type items. Therefore, an automatic scoring would appear to be a favorable solution with respect of both the introduction of more supply-type items on high-stakes standardized tests and on the lower stakes classroom-instruction environment.

Paper-based versus Image-based Scoring of Student Essays

김인숙,박태준 한국응용언어학회 2011 응용 언어학 Vol.27 No.2

Essays have been incorporated into many of the standardized language testing programs since assessing students’ writing ability directly using essays has many values. However, scoring a large quantity of student essays is costly and time consuming. Paper-based scoring, which refers to scoring of essays on paper as submitted by students, had been the standard practice of essay scoring until image-based scoring, which refers to scoring of scanned images of essays, became available due to advances in computerized scanning technicology. Because various aspects of image-based scoring could make it potentially different from paper-based scoring, causing concerns for the reliability of the assessment made, the current study compared paper-based and image-based essay scoring modes using essays from the Georgia 8th Grade Supplemental Writing Assessment in order to examine the impact of image-based scoring on the reliability of essay test scores. The results showed that paper-based and image-based scoring were similar in terms of reliability, providing evidence for the comparability of paper-based and image-based scoring. Although limited in scope, this study adds to the literature on writing assessment by presenting empirical evidence in support of the transition from paper-based scoring practice to image-based scoring practice.

Paper-based versus Image-based Scoring of Student Essays

Insuk Kim,Taejoon Park 한국응용언어학회 2011 응용 언어학 Vol.27 No.2

Essays have been incorporated into many of the standardized language testing programs since assessing students’ writing ability directly using essays has many values. However, scoring a large quantity of student essays is costly and time consuming. Paper-based scoring, which refers to scoring of essays on paper as submitted by students, had been the standard practice of essay scoring until image-based scoring, which refers to scoring of scanned images of essays, became available due to advances in computerized scanning technicology. Because various aspects of image-based scoring could make it potentially different from paper-based scoring, causing concerns for the reliability of the assessment made, the current study compared paper-based and image-based essay scoring modes using essays from the Georgia 8th Grade Supplemental Writing Assessment in order to examine the impact of image-based scoring on the reliability of essay test scores. The results showed that paper-based and image-based scoring were similar in terms of reliability, providing evidence for the comparability of paper-based and image-based scoring. Although limited in scope, this study adds to the literature on writing assessment by presenting empirical evidence in support of the transition from paper-based scoring practice to image-based scoring practice.

Cacium scoring과 허혈성 심장질환 유병률과의 관계

박형민(Hyung Min Park),이현성(Hyun sung Lee),김성옥(Sung Ok Kim),성봉욱(Bong wook Sung),서재영(Jae Young Suh),이종호(Jong Ho Lee),최남길(Nam Gil Choi) 대한CT영상기술학회 2007 대한CT영상기술학회지 Vol.9 No.1

Purpose Purpose of this study is to know relative relation between Calcium scoring and prevalence rate of Ischemic heart disease, therefore it was divide the dangerous degree, using calcified material Risk classified list in Calcium scoring test. Materials and Methods Chest CT( Cardiac a) underwent from November, 2006 January, 2007. T-score at Calcium scoring examination did 214 (36.58 %) who Test more than 0.1 by target except young child patient, Calcium scoring is a patient who was 0 that have been ever enforced heart and Percutaneous Coronary Intervention(PCI) among 585 patient that is prescribed Chest CT(Cardiac A). Target patient man 115 people (53.7 %), average age 63.2 and woman 99 people(46.3 %), average age 65.8. T-score value that score is gotten by Calcium scoring scan classify, (1)group when it is below 100, 100 to 400 (2)group, and 400 to 1000 (3)group, and more than 1000 to (4)group. And analyzed by back to front about relation of Calcium scoring and ischemic heart disease divides and examined by each LM, LAD, LCX, RCA and compares interpretation sequence with each Coronary arteries T-score value after post enhancement CT scan and find Coronary artery stenosis or Perfusion Defect. Results Calcium score appeared highest by 23,110.5 in LAD and ranged in RCA (18, 996.4), LCX (7, 360.6), LM (3, 412.3) order. Also, when was expose in LAD as Stenosis position distribution chart calcium score’s result, RCA, LCX, LM order and it is 36.6% when is (4)group perfusion defect probability (4)group, and it is 26.3% when is (3)group, it is when is 10% (2)group, it is when is 5.2% (1)group. Conclusion When the Calcium score and ischemic heart disease that analyze data on the basis of diagnostic results, it could know that possibility is high to cause ischemic heart disease, if calcium score was high. Considering these results, position of abnormality or other details information was not agree correctly, It is Calcium scoring scan not only screening test, but also meaning of independent test.

정답 템플릿 작성 방식에 의한 한국어 서답형 문항 자동채점 시스템

박일남(Il-Nam Park),강승식(Seung-Shik Kang),노은희(Eun-Hee Noh),김명화(Myung-Hwa Kim),성태제(Tae-Je Seong) 한국정보과학회 2013 정보과학회 컴퓨팅의 실제 논문지 Vol.19 No.12

본 논문에서는 단어-구 수준의 한국어 서답형 문항을 컴퓨터 기반 채점 방식으로 처리하는 방법을 제안한다. 수천 명의 채점자가 채점 기준표를 보면서 수작업으로 채점하는 방식을 컴퓨터를 이용한 자동채점 소프트웨어로 대치하기 위하여 수동 채점을 목적으로 작성된 채점 기준표를 자동채점을 위한 XML 기반의 정답 템플릿으로 기술한다. 정답 템플릿을 기준으로 정답 일치 채점, 개념 기반 채점, 단서어 기반 오류 처리의 과정을 통해 학생 답안에 점수를 부여하는 자동채점 시스템을 설계하고 개발하였다. 자동채점 시스템의 성능을 평가하기 위해 자동 채점 결과와 실제 채점자가 부여한 점수를 비교, 분석하였다. 그 결과에 의하면 실험에 사용한 11개 문항의 평균 카파 계수는 0.83으로 실제 채점자와 비교했을 때 자동채점 시스템은 수작업 채점과 일치도가 매우 높은 성능을 나타내었다. This paper proposes a computer-based automatic scoring method for Korean short answer questions. The purpose of this research is to replace a traditional method of human raters by scoring guide-line with an automatic scoring by computer software. We transformed a hand-written scoring guide-line to a machine-readable answer template that is an XML format. Automatic scoring system consists of four steps: model answer scoring, concept-based scoring, cueword-based scoring, and manual scoring. Kappa coefficient showed an average 0.83 between automatic and manual scoring method.

Efficacy of Corpus-driven Scoring Methods for Computer-based Dictation Tests

최인철,박성연 한국멀티미디어언어교육학회 2016 멀티미디어 언어교육 Vol.19 No.4

The present research is a follow-up research of an exploratory validation study of computer-based dictation tests utilizing TTS (text-to-speech) (Choi & Park, 2015). The study attempted to explore the efficacy of different scoring methods utilizing corpus linguistic techniques. To this end, the data collected from a prior study was employed in the research. 117 Korean EFL learners participated in this research and were divided into three groups depending on educational levels as well as their proficiency levels: 37 UNI (advanced), 40 HSU (intermediate), and 40 HSL (low). Participants' responses were scored using six different methods: GS (graduated scoring), AM (appropriate meaning-based scoring), EW (exact-word scoring), WM (wordlist match scoring), WML (wordlist match scoring including lemmas), and WMS (wordlist match scoring including synonyms). Comparative analysis of the efficacy of the scoring methods revealed that the four methods were highly correlated with each other and that corpus-based wordlist matching was found to be the most effective and robust scoring method. Among the three wordlist match scoring methods producing robust results, WM was proven to be the most effective semi-automated method with high practicality, requiring the least amount of time and energy for scoring.

랜덤포레스트 알고리듬을 활용한 한국어 에세이 자동채점 모델 개발 연구

박종임,최숙기,박종임 한국교원대학교 뇌·AI기반교육연구소 2023 Brain, Digital, & Learning Vol.13 No.2

In this study, Random Forest algorithm was applied using data of 402 Korean argumentative essays to develop an automatic scoring model for Korean essays. Recent studies on automatic scoring of essays are developing into studies using deep learning-based algorithms. However when automatic scoring is used in the field of education, deep learning-based algorithms have limitations because the interpretation and explanation of the scoring results must be possible. Therefore, in this study, we tried to develop a model that can interpret the scoring results by applying a machine learning-based algorithm that utilizes scoring features. In the essay, various parts of morpheme-based scoring qualities were derived, and the number of sentences was used. As a result, it was possible to derive a significant level of model performance. However, this study has a limitation in that it utilized basic quantitative feature such as parts of morpheme frequency and number of sentences. In future research, more in-depth grading feature will be developed to explore ways to provide more meaningful educational feedback through automated essay scoring.

Reader Expectations and the Validity of Composition Scoring: Implications for Componential Scoring

Jungok Bae(배정옥) 한국영미어문학회 2007 영미어문학 Vol.- No.84

  The validity of writing assessment has greatly improved since performance-based writing tasks have been used as alternative writing tasks. High reliability of scoring has also been demonstrated in numerous essay assessments. A relatively recent dimension of validity that has been called into question in writing assessment is scoring validity. This paper discusses scoring validity, focusing on the question of diverse reader expectations that contribute to doubts about scoring validity. Reader expectations are discussed in categories such as: differing reader views of what is good writing, issues of content conceptualization, characteristics of texts and their effects on readers, and divergent backgrounds and assumptions held by writers and readers. The paper suggests that readers in essay assessments should be aware of these issues. Finally the paper finds implications for scoring validity in holistic and componential scoring and recommends componential scoring as an alternative to holistic scoring where appropriate and feasible.

Rating of Korean Students’ L2 Writing : Similarities and Differences Between Native and Non-Native Raters

Lee, Seongyong,Choe, Hohsung 이화여자대학교 교과교육연구소 2012 교과교육학연구 Vol.16 No.3

The present study investigates the rating behavior of two native-speaking teachers and two Korean teachers of English when assessing forty essays written by Korean EFL students. Three research questions guided this study: (1) are there any similarities or differences in their rating behavior in holistic scoring? (2) Are there any similarities or differences in their rating behavior in analytic scoring? (3) And what factors are involved in these similarities or differences? Forty college students preparing for their TOEFL were asked to write an essay. Their essays were evaluated by the four teachers based on the TOEFL writing scoring rubric using the six point scale. The teachers read each essay twice. They were first required to give overall scores; then, in the second reading, they provided analytic assessments for four subareas. A paired t-test was performed to examine any discrepancy in holistic and analytic scoring between the two groups. Differences in holistic scoring appeared statistically significant: the Korean teachers gave higher scores than did their native counterparts. With respect to analytic scoring, the Korean raters also gave higher scores in all subareas, and the differences between the two groups were much greater than those in the holistic scoring. Interestingly, both groups gave the highest score in vocabulary, followed by grammar, content, and organization. A qualitative analysis indicated that all the teachers prioritized the logical flow of ideas and emphasized rhetorical features rather than linguistic features. Native teachers tended to score linguistic features based on native intuition. They also followed the Western writing convention in rating, while the Korean teachers applied the Korean rhetorical style. 본 논문은 원어민 교사와 한국인 교사의 영어작문평가 행위를 연구한 것이다. 원어민 교사 2명과 한국인 교사 2명이 연구에 참여하였으며, 이들은 미국 학부과정으로 유학을 준비 중인 대학생 40명의 토플 작문에 대하여 통합적 및 분석적 평가를 실시하였다. 평가는 토플작문평가 기준표를 사용하여 0에서 5점 사이의 점수를 부여하는 방식이었다. 통합적․분석적 평가에 있어서 두 집단 간 차이를 보기 위해서 t-검증을 실시하였다. 통합적 평가에서 한국인교사들이 원어민교사들 보다 높은 점수를 주었으며, 이는 통계적으로 유의미하였다. 분석적 평가에서도 한국인 교사들이 더 높은 점수를 부여하였고 그 차이가 통합적 평가보다 훨씬 큰 것으로 나타났으며, 이 역시 통계적으로 유의미한 것이었다. 분석적 평가의 항목별 점수를 보면, 두 집단 모두 단어, 문법, 내용, 구성의 순으로 높은 점수를 주었다. 정성적 분석에서는 모든 교사들이 공통적으로 논리의 흐름을 중시하고 내용과 구성을 강조한 것으로 나타났다. 차이점으로는 한국인 교사들은 단어사용과 문법 평가에 어려움을 겪은 반면, 원어민교사들은 직관에 따라 상대적으로 쉽고 정확하게 평가하는 것으로 나타났다. 또한, 원어민 교사들은 영어작문이 요구하는 수사학 형식에 따라 채점한 반면, 한국인 교사들은 채점 과정에서 한국어 작문의 수사학 형식을 따르는 경향이 있었다. 교사들의 언어․문화적 배경과 교육적 배경이 영어작문 채점에 영향을 끼치는 것으로 나타났다.