Predicting the Item Difficulty of a Simulated CSAT English Test Based on Corpus Analysis

엄혜랑 한국영어평가학회 2021 영어평가 Vol.16 No.1

This study investigates the relationship between linguistic features and item difficulty of a simulated College Scholastic Ability Test (CSAT) English test, based on a corpus analysis. The test used for the present study was a simulated CSAT English test administered in June 2020. Item difficulty data was collected from 101,386 students who took the test. For the corpus analysis, lexical and syntactic variables were measured by the Lexical Complexity Analyzer (LCA) and the L2 Syntactic Complexity Analyzer (L2SCA), the computational tool and were correlated with item difficulty (dependent variable) for 41 test items. According to the correlation analysis, one lexical variable and all syntactic variables were found to be significantly correlated with item difficulty. Also, the results of the multiple regression indicate that lexical sophistication and particular structures are related to item difficulty, explaining for approximately 55.1% of the variance in item difficulty. The results showed that the variables identified in the current study were explanatory in terms of predicting item difficulty of the CSAT English test. Therefore, the findings of this study have pedagogical implications for test developers and education policy makers in Korea, with regard to text characteristics and test difficulty. This study investigates the relationship between linguistic features and item difficulty of a simulated CSAT English test, based on a corpus analysis. The test used for the present study was a simulated College Scholastic Ability Test (CSAT) English test administered in June 2020. Item difficulty data was collected from 101,386 students who took the test. For the corpus analysis, lexical and syntactic variables were measured by the Lexical Complexity Analyzer (LCA) and the L2 Syntactic Complexity Analyzer (L2SCA). The computational tool and were correlated with item difficulty (dependent variable) for 41 test items. According to the correlation analysis, one lexical variable and all syntactic variables were found to be significantly correlated with item difficulty. Also, the results of the multiple regression indicate that lexical sophistication and particular structures are related to item difficulty, explaining for approximately 55.1% of the variance in item difficulty. The results showed that the variables identified in the current study were explanatory in terms of predicting item difficulty of the CSAT English test. Therefore, the findings of this study have pedagogical implications for test developers and education policy makers in Korea, with regard to text characteristics and test difficulty.

A Corpus-based Analysis of the Linguistic Complexity Levels of the Reading Passages in Three Asian EFL Countries' College Entrance English Exams

엄혜랑 한국영어평가학회 2023 영어평가 Vol.18 No.1

This study investigates the linguistic complexity levels of the reading passages of the foreign language (English) domain of the College Scholastic Aptitude Test (CSAT) in Korea and compares them with those in two other Asian EFL countries. For the study, the reading passages of the Korean, Chinese, and Japanese college entrance English exams from the 2018 to 2020 academic year were collected and analyzed based on the length, propositional content and organizational characteristics by using a variety of corpus linguistic tools. The Kruskal-Wallis test was administered in order to see if there was a statistically significant difference among the tests. Also, when between-group differences were detected, the Mann-Whitney U post hoc test was conducted to further look into significant differences between the CSAT and the other tests. Overall findings suggest that the reading passages of the CSAT English exam maintained the highest difficulty level among the three Asian EFL countries’ college entrance English exams, with cognitively demanding academically-oriented items, highly advanced vocabulary, complex syntactic structure and low readability. Based on the findings, this research is expected to shed light on the intention of incorporating criterion-referenced English assessment into the CSAT as well as provide pedagogical implications for test developers and education policy makers in Korea.

국내 대학생의 우울에 관한 연구 동향 : 국내 학술지 논문을 중심으로

엄혜인(Um Hye In) 단국대학교 상담학연구소 2019 상담교육연구 Vol.2 No.1

본 연구는 국내 대학생의 우울에 관한 동향 연구이다. 이를 위해 한국교육학술정보원(KRIES)의 RISS에서 KCI 등재지 또는 등재 후보지에 게재된 주요 학술지 논문 중 ‘대학생의 우울’과 관련된 논문 총 49편을 최종 연구 대상으로 선정하였다. 분석 기준을 연구 시기, 연구 변인, 측정 척도, 연구 방법 및 통계 분석으로 나누어 살펴보았다. 연구 결과 대학생의 우울에 관한 연구는 2010년에 처음 등장하여 매년 최소 2편 이상의 연구가 진행되었다. 변인별 동향은 종속변인 39편, 독립변인 7편, 매개변인 4편에 대학생의 우울이 사용되었고 대학생의 우울이 종속변인으로 사용되었을 때의 독립변인은 심리적 요인이 많았다. 측정 척도로는 역학연구센터 우울 척도(CES-D), Beck의 우울 척도(BDI), Beck의 우울 척도-Ⅱ(BDI-Ⅱ)가 주로 사용되었다. 연구 방법으로는 양적연구가 압도적으로 많이 되었고 통계 분석으로는 회귀분석이 대다수를 차지하였다. 이와 같은 결과를 토대로 본 연구의 한계점과 향후 대학생의 우울에 관련된 연구의 시사점을 제시하고자 한다. Objectives: This study is about trends in depression among university students in Korea. Methods: The RISS of Korea Institute of Education and Research Information (KRIES) selected a total of 49 papers related to the university students’ depression among the major academic journals published in KCI registrations or candidate registrations. The analysis criteria were divided into study timing, study parameters, measurement scales, study methods, and statistical analysis. Results: Journal of Counseling and Research Published by DKU Counseling Research Institute. Conclusions: The study of college students depression first appeared in 2010, and at least two studies were conducted each year. The trends by variables used college students depression in the 39th episode, the 7th episode of independence, and the 4th edition of parameters. There were many psychological factors when college students depression was used as a dependent variable. The measures were mainly used by the Center for Epidemiological Studies (CES-D), Beck s depression scale (BDI) and Beck s depression scale-II (BDI-II). Quantitative research was overwhelmingly conducted by research methods, and the majority accounted for statistical analysis. Based on these results, I would like to illustrate the limitations of this study and the implications of the study related to university students’ potential depression in the future.

Exploring the Efficacy of ChatGPT as a Language Learning and Assessing Tool

엄혜랑(Hye Rang Om) 한국영어평가학회 2024 영어평가 Vol.19 No.1

This study aims to assess how well ChatGPT-3.5 performs on the English reading comprehension section of the College Scholastic Ability Test (CSAT, henceforth), with a focus on its viability as a tool for language learning and evaluation. To do this, tests were conducted where ChatGPT answered questions from the 2022 to 2024 CSAT English exams, and its accuracy was recorded. The research also looks into how ChatGPT’s performance changes depending on the type of question and the version of the AI. The results showed that ChatGPT-3.5 correctly answered about 74.1% of the questions, whereas GPT- 4 had nearly flawless accuracy. The research outlines the capabilities and shortcomings of the large language model (LLM) and discusses areas that require enhancement for educational use. Furthermore, the research examined if these trends hold true for other international English language tests, specifically using reading questions from the TOEFL iBT. Similar performance trends were observed with the TOEFL iBT. The findings indicate that the LLM’s abilities in reasoning and explanation could benefit language education. However, it also emphasizes the necessity for ChatGPT to acquire more specific knowledge and improved reasoning abilities to understand context better.

Synthesis and Characterizations of Various MnO2 for Rechargeable Lithium-Air Battery

엄혜리,김태윤,김민경,백성현 한국공업화학회 2012 한국공업화학회 연구논문 초록집 Vol.2012 No.-

고전검사이론과 문항반응이론으로 살펴본 모의 TOEIC 문항분석

엄혜랑(Hye Rang Om),이예나(Yena Lee) 한국영어평가학회 2023 영어평가 Vol.18 No.2

This study analyzed student test performance on the mock TOEIC in order to investigate the validity and reliability of the test items within a framework of Classical Test Theory (CTT) and Item Response Theory (IRT). The test, including 49 listening and 51 reading multiple-choice items, was administered to a total of 98 registered university students. Their responses to each test item were statistically analyzed with respect to four indices: item difficulty, item discrimination, item guessing and distractor’s attractiveness. For the analysis, each item was analyzed by applying CTT and IRT according to a 3-parameter logistic model with Pearson correlation coefficient and Kuder Richardson-20 (KR-20). According to the item analysis, it was found that the test displayed a high level of reliability of .920, and the difficulty level was .563, indicating that it was in the desirable range between .25 and .75. The mean discrimination power was also .368, which is within the desirable range of over .3. However, further statistical and qualitative item analyses found that the mock TOEIC contained test items within an inappropriate difficulty level, low item discrimination and implausible distractors. Therefore, this study provides suggestions for enhancing the quality of the mock TOEIC, which can also apply to future TOEICs.

대학 편입영어시험의 유용성 검증

엄혜랑(Om, Hye Rang) 한국영어평가학회 2015 영어평가 Vol.10 No.1

The purpose of this study is to investigate the components of the Korean college transfer English exam in relation to the 1iterature of test usefulness. In this study, Korean college transfer test of English samples from 2012 to 2014 were selected. Their construct validity from aspects of the test method and test content based on Bachman and Palmer (1996) s framework of test method facets was examined, and problematic items were diagnosed. Then, lexical analysis borrowed from corpus linguistics was conducted. Then the exam was analyzed in terms of Flesch Reading Ease levels. The study, which investigated students perceptions of the Korean college transfer exam of English, was conducted using a questionnaire survey collected from 33 students. The results show that the exam has supported some principles of usefulness, but in reality many qualities of test usefulness were revealed not to be regarded. Suggestions are made on the basis of the results for the improvement of the test in a new format.

절대 평가제 도입 이후 수능 영어 독해지문의 난이도 분석 : 어휘 복잡성과 통사적 복잡성을 중심으로

엄혜랑(Om Hye Rang) 학습자중심교과교육학회 2022 학습자중심교과교육연구 Vol.22 No.6

목적 본 연구는 2018학년도 대학 수학능력 시험 영어영역에 절대 평가제가 도입된 이후 수능 영어 독해지문이 어휘 복잡성(lexical complexity)과 통사적 복잡성(syntactic complexity) 측면에서 일정한 난이도를 유지하고 있는지 분석해 보고자 한다. 방법 이를 위해 2018~2021 수능 영어 독해지문을 대상으로 코퍼스를 구축하였으며 LCA(Lexical Complexity Analyzer)와 L2SCA(L2 Syntactic Complexity Analyzer)를 사용하여 25개의 어휘 복잡성(어휘 밀도, 어휘 수준 정도, 어휘 변화도) 측정치와 14개의 통사적 복잡성(생성 단위 길이, 문장 복잡성, 종속성, 등위성, 특정구조) 측정치로 분석하였다. 그리고 연도별 시험의 난이도 비교를 위해서 일원 배치 분산분석 (One-way ANOVA)을 수행하였다. 결과 연구 결과에 따르면 2018~2021 수능 영어 독해지문은 모든 측정치에서 유의미한 난이도 차이를 보이지 않은 것으로 나타났다. 결론 이상의 연구 결과는 절대 평가제 도입 이후 4년 동안 수능 영어 독해지문이 어휘 복잡성 및 통사적 복잡성의 측면에서 난이도의 일관성을 유지하고 있다는 것을 보여준다. 또한 이 연구는 수능 절대평가제와 관련하여 교육적 함의와 시사섬을 제공하고 있다. Objectives The purpose of the present study is to investigate whether the CSAT(College Scholastic Ability Test) English reading passages are consistent in the level of difficulty on the basis of the criterion-referenced assessment introduced in the 2018 academic year. Methods To achieve this goal, English reading passages from the 2018-2021 academic years were collected and analyzed with 25 measures for lexical complexity(lexical density, lexical sophistication, and lexical variation) and 14 measures for syntactic complexity(length of production, sentence complexity, subordination, coordination, and particular structures). In order to see if there was a statistically significant difference among the four tests, one-way ANOVA was performed. Results The results show that there was no significant difference in terms of lexical complexity and syntactic complexity among the CSAT English reading passages during the academic years 2018-2021. Conclusions This suggests that there was no significant yearly variance in the CSAT English sections under the criterion-referenced assessment. Additionally, the findings of this study provides some pedagogical implications and suggestions for the implementation of the criterion-referenced assessment.

환원법에 의한 리튬공기전지용 MnO₂/Carbon composite 제조

정구봉,백성현,엄혜리 한국공업화학회 2010 한국공업화학회 연구논문 초록집 Vol.2010 No.1

Synthesis of Carbon Supported Metal Catalysts for Li-Air Battery

고보경,김태윤,김성환,엄혜리,백성현 한국공업화학회 2012 한국공업화학회 연구논문 초록집 Vol.2012 No.-