A FACETS Analysis of Rater Characteristics and Rater Bias in Measuring L2 Writing Performance

신유선(Yousun Shin) 한국영어어문교육학회 2010 영어어문교육 Vol.16 No.1

The present study used multi-faceted Rasch measurement to explore the characteristics and bias patterns of non-native raters when they scored L2 writing tasks. Three raters scored 254 writing tasks written by Korean university students on two topics adapted from the TOEFL Test of Written English (TWE). The written products were assessed using a five-category rating scale (Content, Organization, Language in Use, Grammar, and Mechanics). The raters only showed a difference in severity with regard to rating categories but not in task types. Overall, the raters scored Grammar most harshly and Organization most leniently. The results also indicated several bias patterns of ratings with regard to the rating categories and task types. In rater-task bias interactions, each rater showed recurring bias patterns in their rating between two writing tasks. Analysis of rater-category bias interaction showed that the three raters revealed biased patterns across all the rating categories though they were relatively consistent in their rating. The study has implications for the importance of rater training and task selection in L2 writing assessment.

코메트릭스(Coh-Metrix)를 활용한 공무원 7급과 9급 영어 시험 지문 분석 : 어휘 다양성, 통사적 복잡성, 응집성과 이독성 지수를 중심으로

신유선(Yousun Shin),정영경(Youngkyong Jong) 한국영미어문학회 2018 영미어문학 Vol.- No.129

This current study aimed at examining the differences between the reading passages used in the English tests for 7 th grade and 9 th grade civil servants in terms of lexical diversity, syntactic complexity, cohesive patterns, and readability. Additionally, the correlations were explored among the 14 indices used in the analysis. The reading passages from three-consecutive years (’14-16) of English tests for prospective civil servants were extracted, compiled into corpora, and analyzed using Coh-Metrix to examine the difficulty of the reading demands of civil servant tests. The results demonstrated that there were no statistically significant differences across all 14 variables between the reading passages extracted from the English test for 7 th and 9 th grade civil servants, respectively. It was found out, however, that there existed some significant positive and negative correlations among the comparable variables. The current study implied that test developers need to carefully calibrate the text difficulty between two different levels of civil servant tests so that each test can be used as a legitimate and fair measurement tool to differentiate test-takers by their selected grade. The implications and the limitations of the current study were also provided.

공무원 영어시험의 어휘 부담에 관한 연구

신유선(Yousun Shin),김은지(Eun-Jee Kim),오준일(Junil Oh) 한국영미어문학회 2017 영미어문학 Vol.- No.125

The study aimed at addressing two questions: (1) What are the vocabulary demands of English tests for prospective ninth-level, seventh-level civil servants, and police officers?; and (2) How consistent and uniformed has the vocabulary levels of such tests been over the years? Texts from three-consecutive-years-examination papers (2014, 2015, and 2016) were compiled into corpora and analyzed using VocabProfile to find out the vocabulary demands of the public servant examinations. On average, ninth-level civil servants’ English exams were found to use 2,139 tokens, and seventh-level exams used 2,114 tokens, while police officers’ exam used 1,409 tokens. In order to reach 95% text coverage, test takers were required to know 3,882, 3,479, and 2,890 word families for ninth-level civil servants’, seventh-level civil servants’, and police officers’ exams, respectively. Additionally, to reach 98% text coverage, they should obtain substantially higher knowledge of word families which were found to be the case for 7,497, 6,219, and 5,366 for ninth-level, seventh-level civil servants, and police officers, respectively. The results showed that inconsistency was found in the vocabulary levels for each test across three consecutive years. It was also suggested that there was no clear distinction found in the levels of vocabulary between ninth-level and seventh-level civil servants’ tests, which might be a factor compromising the validity of the high-stakes tests to some extent. The implications of the findings are discussed.

래쉬 모델을 활용한 예비교사들의 영어 글쓰기에 대한 불안 및 불안 이유 분석

신유선(Yousun Shin) 팬코리아영어교육학회(구 영남영어교육학회) 2022 영어교육연구 Vol.34 No.2

The main purpose of this study was to examine L2 writing anxiety among pre-service teachers in English education and the reasons for L2 writing anxiety. A total of 163 respondents completed three different questionnaires online. The research questions included: (1) How valid is the L2 writing anxiety questionnaire to measure writing anxiety? (2) Is there a significant difference between the participants' writing anxiety in L1 and L2? (3) Are any relationships between the subcategorized variables of L2 writing anxiety, L1 writing anxiety, and reasons for L2 writing anxiety present? As the result of the analysis, the L2 writing anxiety questionnaire proved to be valid and reliable when analyzed through a Rasch model. The results of the ANOVA indicated that English pre-service teachers showed significant differences in L1 and L2 writing anxiety. The participants were more anxious under the L2 context (somatic anxiety, cognitive anxiety, and avoidance behavior) than they were under the L1 context. Lastly, correlational analysis among all variables revealed that L2 writing anxiety and reasons for L2 anxiety were positively correlated (attitude toward the writing task, use of writing strategies, textbook materials, writing competence, the impact of a writing instructor, prior writing experience).

키워드 네트워크 분석을 활용한 영어교육관련 연구 동향 분석 : 2000-2019년도까지 「영어교육연구」에 게재된 논문을 중심으로

신유선(Shin, Yousun),김양희(Kim, Yanghee) 팬코리아영어교육학회(구 영남영어교육학회) 2020 영어교육연구 Vol.32 No.4

The purpose of this critical review of the literature is to identify the characteristics of keywords between two periods (2000-2009 and 2010-2019) and relations among the research keywords. This study conducted a series of keyword analysis of the articles published by English Language Teaching for 20 years. A total of 814 (2000-2009: 412, 2010-2019: 402) research articles were analyzed in terms of publication year, author, and keywords. The results using a data-mining software, Netminer4, demonstrated that most of main research topics were in tandem with theoretical backgrounds and applicable concepts within the communicative language teaching approach. Common research topics from the frequency analysis included ‘communicative competence’, ‘motivation’, ‘reading comprehension’, ‘teacher education’, and ‘listening comprehension.’ One of the distinctive features between two periods was that in the 2000-2009 period research interest was on the application and utility of multimedia in English learning while in the latter period, a considerable number of studies highlighted internal factors affecting language learning, such as learner motivation and learner perception. Additionally, the analysis of keywords based on centrality analysis revealed that the extension of the research topics and the close association among those topics has been gradually prominent through 20 years. (195 words)

어휘와 문법지식이 개별 출력중심 딕토글로스 과제 수행에 미치는 영향 : 대학교 EFL 학습자를 대상으로

신유선(Shin, Yousun) 팬코리아영어교육학회(구 영남영어교육학회) 2021 영어교육연구 Vol.33 No.2

The objectives of this study are mainly two-fold. First, it is to analyze the cohesion characteristics, vocabulary, and grammar knowledge of the writing tasks, which English learners produced using dictogloss protocols. The second aim is to identify the relative contributions of lexical and grammatical knowledge to the writing products yielded by the participants. For this purpose, 45 English learners at a university completed four different tests and then completed two types of dictoglass tasks with two different topics over four weeks. The collected data were analyzed using a web-based analysis software called Coh-Metrix. The results of the analysis indicated that participants in the group with higher vocabulary scores yielded higher mean scores in all six dependent variables within three categories of fluency, lexical complexity, and grammatical complexity. Of six dependent variables, three measures of Total Word Count, Latent Semantic Analysis, and Average Sentence Length showed statistically significant differences in group comparisons. No statistically significant results, however, was found regarding the category of lexical complexity. With relative contributions of lexical and grammatical knowledge, both vocabulary tests were analyzed as significant factors affecting learners’ task performance. However, only one grammar test - error identification test - was analyzed as a variable that affected task performance. (203 words)

Investigating variability in tasks, analytic scoring criteria, and rater groups on L2 writing in English

Yousun Shin(신유선),Youngkyong Jong(정영경),Yanghee Kim(김양희) 한국외국어교육학회 2010 Foreign languages education Vol.17 No.2

The purpose of the study is to investigate how the judgment of raters is biased toward certain rating criteria or task types in assessing Korean university students’ L2 writing. In addition, the study aims to examine whether or not non-native rater group and native rater group share the same judgment of students’ written products. Six raters scored 258 compositions written by Korean university students on two topics. The written products were rated using an analytic scoring scale with content, organization, language in use, grammar, and mechanics. For the first part, the FACETS analysis revealed that all six raters showed self-consistent scoring patterns across the rating categories in the two tasks. However, each rater was biased towards certain type of tasks and rating categories. For the second part of the study, the results of ANOVA and MANOVA tests indicated that there were significant mean differences between the two groups across the rating categories in both tasks except content in Task 2. It is shown that the non-native rater group is harsher in their rating than the native rater group regardless of the rating categories.

The Effects of Pre-task and Post-task Activities on L2 Vocabulary Retention

Park, Si Hyoung(박시형),Shin, Yousun(신유선) 새한영어영문학회 2017 새한영어영문학 Vol.59 No.1

The study aims to explore the effects of pre-and-post task activities on vocabulary retention in L2 learners in Korea. The 64 university students participated in the study and they were randomly assigned to one of the two conditions (pre-task group and post-task group). To measure their lexical retention, two main tests were administrated: a Vocabulary Size Test (VTS) and a vocabulary test (immediate and delayed test) which was designed by a researcher. A series of MANOVA and repeated-measures were used to analyze the data collected for this study. The results suggested that significant mean differences were found in task types and proficiency levels but the interaction effect between task types and proficiency levels was not found. In summary, the participants tended to obtain the higher scores in both immediate and delayed vocabulary tests when they were involved in the post-task activity. The results also revealed that the advanced learners’ performance was superior to that of the novice group learners regardless of task types, which implies that the degree of attrition on vocabulary learning tended to slow down for the advanced group learners in comparison to that of the novice group learners. Pedagogical implications and future research directions are further suggested.

Comparative Analysis of Post-editing Techniques for Chinese-to-English Translation Tasks: A Quasi-Experimental Study

Chen Xu(Chen Xu ),신유선(Yousun Shin) 한국영미어문학회 2023 영미어문학 Vol.- No.149

This study aimed to examine the differences in translation time and quality between human translation (HT), machine translation post-editing (MTPE), and ChatGPT post-editing (ChatGPT PE). A quasi-experimental design was employed, involving 30 junior participants majoring in English from a Chinese university. A translation task consisting of 532 words from Chinese to English was assigned to the three groups and the translation speed and quality were assessed using the Multidimensional Quality Metrics (MQM) and Dynamic Quality Framework (DQF). The results demonstrated that the post-editing technique could produce translation quality comparable to human translation while achieving faster translation speed. It was observed that ChatGPT PE outputs exhibited the highest number of terminology errors but had the fewest accuracy errors, while mean differences in two of accuracy dimensions were observed among the three groups. The utilization of post-editing in language teaching would allow for the integration of machine translation strengths and human expertise, resulting in high-quality translations on par with human translations. To optimize post-editing outcomes, it is crucial to develop and teach error-detection techniques and avoid excessive reliance on AI technology.

A Trajectory for Faculty Professional Development in Collaborative Course Design

박은정(Eunjeong Park),신은영(Eun Young Shin),신유선(Yousun Shin) 한국영미어문학회 2021 영미어문학 Vol.- No.143

This study explored faculty professional development through a specific course design at a medium-sized national university in South Korea. Three professors in an English education major participated in the project-based faculty professional development to design a course and plan to create course content. Data were collected through interviews, meeting agenda notes, faculty reflection logs, and relevant documents during the fall semester of 2021. Thematic analysis was used to identify emerging themes or incidents regarding issues from the faculty professional development. During the faculty professional development with designing a new course, four intriguing themes were identified for promoting faculty professional development and strengthening a faculty community of practice; (1) creating an adequate atmosphere for faculty professional development, (2) sharing various perspectives with cultural competence, (3) negotiating and accommodating the direction of course design and related research and (4) awareness of constraints of time management. Commitment to constant communication and course design collaboration offers positive pedagogical implications for faculty professional development.