Detecting Item Bias Using an Item Response Theory(IRT) Model

Chi, Eunlim 경희대학교 부설 교육문제연구소 1997 論文集 - 경희대학교 교육문제연구소 Vol.12 No.1

This study examined item bias occurring in a. language test which administered to two groups. The two groups could be divided into Koreans and Americans based upon their first Language background. The language test used in this paper was Word Analysis of ITBS. This intended to examine item bias caused by the background of the first language. Item bias in the language test was explored by using an item response theory model. Specifically, Rasch model which was one-parameter logistic model among item response theory models was applied to calibrate items. The difficulty measures of items which were calibrated by the Rasch measurement helped to obtain the accurate comparison of groups by the invariance of item difficulty measures. In calibrating items, the computer program MSCALE was used. The two groups were compared for item difficulty measures which were obtained by MSCALE. The comparison was made by the computer program IPARM using chi-square statistics. The results identified five items as having significantly large group differences. That is, the five items had different difficulty measures for the two groups. The distractor analysis and the content analysis for the five items revealed that the linguistic background of the two groups affected item difficulties. This suggests that items can be biased against a group due to the linguistic backgrounds of groups. These results showed that item bias caused by ethnic characteristics could be well identified by using an item response theory model. By the objective measures provided by an item response theory model, item bias can be more accurately identified, This means that an item response theory model can be a useful technique for examining item bias. Many techniques for investigating item bias are developed, but it was a problem that they use raw scores. When raw scores are used, the findings about item bias can depend on samples or items used. Therefore, the application of item response theory is required for the objective finding. In addition, the results demonstrated how the factors causing item bias could be explained. The explanation of the factors is more meaningful in studying item bias since the explanation of the factors can help to understand the group characteristics. The understanding of group characteristics can improve teaching and learning for the group. The examination of item bias is required not only for making tests fair but also for improving teaching and learning.

우리나라 대학입시 과열경쟁 현상에 대한 인식 조사

지은림(Eunlim Chi) 한국교육평가학회 2004 교육평가연구 Vol.17 No.2

본 연구는 우리나라 대학입시 과열경쟁 현상에 대해 학생, 학부모, 교사, 평가전문가들이 어떠한 인식을 갖고 있는지를 탐색, 비교해보았다. '2002년 무시험전형제도'와 '2005년 과목선택제 도입'에 대한 인식을 조사해본 결과, 교사를 제외하고 대부분이 공통적으로 이러한 변화들이 본래의 목적은 달성하고 있지만 과열경쟁 해소에는 도움이 안된다고 하였다. 한편, 네 집단 모두 과열경쟁의 주요 원인은 '대학 서열화'라고 생각하며, 전형자료 중 수능과 내신이 과열경쟁을 유발하는 자료인 반면에 논술고사, 면접, 인적성검사는 과열경쟁 유발의 심각성이 비교적 덜한 것으로 보고 있었다. 과열경쟁을 해결하기 위해서는 전인적 품성 평가 및 종합적인 평가가 바람직하며, 전형자료로는 면접이 과열경쟁과 가장 관련이 적은 것으로 보고 있다. 그러나 전문가 집단은 전인적 품성평가나 종합적인 평가가 과열경쟁 해소에 별로 도움이 안될 것으로 보는 비율이 높았다. 과열경쟁 해소를 위한 시급한 과제로 학력위주의 사회제도 개선을 들고 있으며, 변화주체로는 전문가들이 수험생 및 학부모를 생각하는 반면, 나머지는 교육인적자원부를 들고 있었다. 2008년 입시제도안에 대해서는 전문가집단의 인식조사만 실시되었는데, 역시 과열경쟁 해소에는 별로 도움이 되지 못할 것으로 보는 경향이 강하게 나타났다. This study compared the perception of students, parents, teachers, and experts about the excessive competition for university admission in Korea. The survey items included the causes for the excessive competition for university admission, the evaluation of university admission policies and assessment materials, the method for reducing the competition for the univeristy admission and so on. The results revealed that the social-cultural factor was more serious than the admission system or materials for the excessive competition for university admission. Specifically, the ordering of universities was the most serious factor for the excessive competition. As the assessment materials, College Scholastic Aptitude Test(CSAT) was the most serious factor for causing the excessive competition, and the next one was a high School Activities Record(SAR). Meanwhile, it was suggested that the efforts to improve the social system based on the academic background were required to solve the problem of the excessive competition for university admission. These imply that efforts should be made to improve not only the policies for admission systems and assessment materials but also social-cultural systems and atmospheres.

문항반응이론을 적용한 정보화마인드 척도 개발 및 초, 중등 학생들의 수준 평가

지은림 ( Eunlim Chi ) 한국교육공학회 2003 교육공학연구 Vol.19 No.1

교사의 형성평가 피드백 수행 척도 개발 및 타당화

지은림(Chi, Eunlim) 한국교육평가학회 2010 교육평가연구 Vol.23 No.1

본 연구는 형성평가를 위해 사용될 수 있는 교사의 피드백 수행 척도를 개발하기 위해 초등 교사들에게 질문지를 실시했다. 여기서 피드백 수행 문항들은 평가목적, 평가수행, 피드백수행, 피드백유형, 피드백효과에 관한 내용들을 포함했다. Rasch 부분점수모형 분석을 실시한 결과 첫째, 총괄평가와 관련된 평가목적(서열매김, 기준비교) 문항들은 부적합한 것으로 나타났다. 둘째, 차원성 분석과 측정치 비교를 통해, 척도가 태도관련 문항들(평가목적, 피드백유형, 피드백효과)과 수행관련 문항들(평가수행, 피드백수행)로 구분될 수 있는데, 실제 수행 수준을 나타내는 문항들의 측정치가 낮은 경향을 보였다. 셋째, DIF 분석은 교사의 연령대에 따라 평가와 피드백 수행의 차이를 보여주었다. 마지막으로 준거타당도 검증은 개발된 척도가 피드백 수행 수준 파악을 위해 적절하다는 것을 확인시켰고, 신뢰도도 .810으로서 높은 편이었다. This study develops the scale of teachers' feedback practice for formative assessment by applying the Rasch model-Partial Credit Model. The questionnaire used for this study consisted of 28 items(3 background items, 3 criterion-related variable items, 22 feedback practice items). The feedback practice items were related to assessment goal, assessment frequency, feedback frequency, feedback type, feedback effects and had the format of 5 point scale. The questionnaire was administered to 240 primary school teachers. Item fit analysis found 2 misfitting items which were related to summative assessment goals(relative ordering and comparing to standards). The scale developed after deleting the two misfitting items was validated with the criteria variables of ‘feedback practice level’, ‘assessment practice level’, and ‘the degree of thinking assessment practice as teachers’ routine task. The reliability of the scale was also found to be satisfactory(α=.81). Meanwhile, this study revealed several findings for feedback practice. First, teachers tend to have the lower level in practicing assessment and feedback than in their attitude or perception of feedback. They used to prefer marking or scoring to comments as feedback methods, while comments are more recommended than other methods to attain the effect of feedback. The level of practicing assessment and feedback for group written activities was also much lower than others(test, homework, quiz, individual written activities). DIF analysis showed that 20s' and 30's teachers tended to practice more assessment and feedback than elder teachers. As a consequence, the scale developed in this study can be helpful to diagnose the level of teachers' feedback practice and understand the features of feedback practice.

국제비교평가 검사들의 국가간 차이 분석을 위한 Rasch 모형 적용

지은림(Chi, Eunlim) 한국교육평가학회 2011 교육평가연구 Vol.24 No.1

국제화 추세와 함께 많은 심리검사들이 국제비교평가에 사용되고 있으나, 이러한 검사들은 국가들이 가지고 있는 다양한 사회․문화적 배경과 언어의 영향을 제대로 고려하지 못하고 있어 측정학적 오류의 유발에 대한 우려가 커지고 있다. 따라서 본 연구는 여러 국가들을 대상으로 동시에 실시되는 검사에 대해 측정의 타당성과 국가 간 차이에 대해 탐색해보고자 한다. 이를 위한 자료로는 TIMSS 수학 8학년 교사 설문지에 대한 한국, 대만, 미국 교사들의 응답을 사용하였다. 측정모형으로는 Rasch 모형의 확장형태인 부분점수모형(Partial Credit Model)을 사용하였는데 첫째, 검사 특성 및 타당성을 확인하기 위해‘문항 적합도’와 ‘문항과 피험자 추정치 분포’에 대한 분석을 실시했다. 둘째, 국가 간 차이를 탐색하기 위해서 문항 편파성 분석을 실시하였다. 마지막으로 편파성이 발견되는 문항을 조정하여 척도를 재개발 하는 방법을 실시하였다. 본 연구에서 보여준 국가 간 차이 분석을 위한 방법론적 탐색은 향후 국제비교평가에서 사용되는 검사들의 타당성을 확보하고, 편파성을 제거 또는 조정함으로써 국가 간에 동등성을 확보하는데 도움을 줄 수 있을 것으로 기대된다. This study recognizes the seriousness of the equivalence of tests administered across countries and tries to demonstrate how to explore cross-culutural assessment. For the study, TIMSS teachers' survey data for 8th grade math subject were used to compare Korea, Tiwan, and USA. The measurement model applied to the data analysis was partial credit model which was the expanded form of Rasch model, and WINSTEPS, the scaling computer program using partial credit model, was used to examine item fit, item and person measures and to find and adjust DIF items. The result revealed that there were significant differences between the group of Korea and the group of Tiwan and USA in teachers' assessment practice. Korean teachers showed lower levels in most of items. This implies that Korean math teachers less practice assessment than the other two countries. For example, item 7 (“students' outcomes of homework are used for class discussion”) showed the largest differences between Korea and the other two countries. For the item 7, re-scaling was performed treating the item as two different items; one for Korea and the other for Tiwan and USA. That is, teachers' responses were split into two groups(Korea and Tiwan-USA) and estimated again. It was found that the measure of item 7 became far lower for Korea than the other two countries. This analysis tried to demonstrate that the differences between countries could be used to adjust item measures and improved the equivalence of tests across countries by controlling item biases. However, the further studies are suggested to make the goal of exploring the equivalence of cross-cultural assessment explicit. Also, the methods of adjusting DIF items to improve the equivalence of tests should be more examined and compared.

창의적 인재 역량 측정을 위한 구인 탐색 및 척도 개발

지은림(Chi, Eunlim),주언희(Ju, Unhee) 한국교육평가학회 2012 교육평가연구 Vol.25 No.1

본 연구는 창의적 인재 역량의 구인을 탐색하고 측정을 위한 척도의 개발을 목적으로 하여, 중․고등학생 532명을 대상으로 질문지를 실시하고 분석하였다. 창의적 인재 역량을 측정하기 위한 질문지는 인지적(고차적 사고력, 확산적 사고력, 문제해결력), 정의적(호기심, 개방성, 감수성, 과제집착력), 사회적(사회가치추구, 협동 및 배려) 특성의 3개 영역(9개 요인)으로 구성하고 리커트식 5점 척도의 형태로 실시되었다. 자료 분석을 위해서는 Rasch 모형에 의한 문항 적합도 분석과 문항들의 측정치 비교를 실시하였고, 탐색적 요인분석과 함께 구조방정식모형을 적용한 확인적 요인분석이 수행되었다. 연구 결과는 첫째, 창의적 인재 역량은 위와 같은 3개 영역(9개 요인)으로 구성되어 있다는 것을 타당화시켜 주었다. 둘째, 학생들은 영역들 중에서 인지적 영역의 특성이 가장 부족한 편인 반면에, 사회적 영역의 특성이 가장 발달된 것으로 나타났다. 인지적 영역에서도 학생들은 확산적 사고력과 고차적 사고력의 특성이 가장 부족한 것으로 나타났기 때문에, 이에 대한 개발이 필요하다는 것을 알 수 있었다. 셋째, Cronbach-α에 의해 본 연구에서 개발된 질문지의 신뢰도가 검증되었다. 결론적으로 본 연구 결과들은 창의적 인재 역량의 구인을 명료화하고 요인별 수준 차이를 비교해줌으로써, 효과적인 교육 프로그램 개발에 도움을 줄 수 있을 것이다. This study aims to explore the construct of creative leader competency and develop the scale for the trait. For this study, a questionnaire including three domains(cognitive, affective, social) was administered to 532 secondary school students. To analyze the data, item fit analysis and person-anditem map were made with WINSTEPS which was based upon Rasch model. Also, exploratory factor analysis and confirmatory factor analysis were done with the Structural Equation Model. The results, first of all, show that creative leader competency consists of three domains of cognitive, affective, and social characteristics. Here, cognitive domain includes ‘higher mental thinking’, ‘divergent thinking’, and ‘problem solving’, while affective domain covers ‘curiosity’, ‘openness’, ‘sensitivity’, and ‘task commitment’. Social domain, which is newly added to the traditional concept of creativity consisting of only cognitive and affective domains, includes ‘the pursuit of social value’ and ‘cooperations and considerations’. Second, the examination of item measures demonstrates that cognitive domain is most hard to endorsed for students. Especially, divergent thinking and higher mental thinking were hard to endorded in the cognitive domain. Third, Cronbach-α proves that the scale developed in this study is reliable. In consequence, the results of this study help to reconceptualize creative leader competency and apply the scale to measure students in school settings.

Rasch 모형을 적용한 검사동등화에서 가교검사를 위한 동형 문항의 활용 가능성

지은림(Chi, Eunlim),박소연(Park, Soyeon) 한국교육평가학회 2011 교육평가연구 Vol.24 No.4

본 연구는 검사 동등화에서 발생하는 문항 노출의 위험성을 줄이기 위한 방안으로 공통 문항으로 구성된 가교검사를 사용하는 동등화에서 공통 문항의 단일 세트를 사용하는 대신에, 동형의 유사한 문항 세트들을 공통 문항으로 활용할 수 있는 가능성을 검증하고자 했다. 검사 동등화를 위해 대학생 244명을 대상으로 실시된 적성검사 문항들을 가지고 두 개의 동등화 설계를 구성하였다. 〈설계 Ⅰ〉은 동일한 문항들로 구성된 가교검사를 사용하는 동등화인 한편, 〈설계 Ⅱ〉는 동형의 유사한 문항 세트 2개를 공통 문항으로 취급하여 동등화하는 방안이었다. Rasch 모형 공통문항 비동등집단 설계에 의해 검사 동등화를 〈설계 Ⅰ〉과 〈설계 Ⅱ〉에서 각각 실시한 후에, 두 가지 설계에 의한 동등화 결과들을 비교해보았다. 비교 결과는 〈설계 Ⅰ〉과 〈설계 Ⅱ〉가 모수치 추정과 문항 적합도에 있어서 유의미한 차이를 보이지 않는다는 것을 보여주었다. 결론적으로 본 연구는 동형의 문항들을 활용하여 공통 문항 동등화를 실시함으로써 문항 노출의 위험성을 줄일 수 있는 가능성을 제시해주었다. This study aims to confirm the possibility of using equivalent form items, instead of using the exactly same common items, for the anchor test in test equating in order to reduce the risk of exposing test items. For the study, a data set of an aptitude test which was administered to 244 university students was divided into two equating designs. The design I uses the same items, while the design II used the equivalent form items for the anchor test. Test equating was conducted separately for each design by applying the method of Rasch common item nonequivalent group. The outcomes of equating by two designs were compared in terms of the estimation of parameters (item difficulty and person ability) and item fits. The results showed that there was no significant difference between two designs for the estimation of parameters and item fits. The findings suggest that tests can be equated by constructing an anchor test with equivalent form items for the security of test items. Furthermore, the possibility of using equivalent form items is expected to enhance the efficiency of equating tests.

대학 교수자를 위한 강의자가진단도구 개발의 타당화 및 관련변인 분석

지은림(Chi Eunlim),장정아(Chang Jung-A) 한국교육평가학회 2006 교육평가연구 Vol.19 No.1

본 연구는 대학 교수자가 강의를 스스로 평가하고 성찰해볼 수 있는 기회를 제공하기 위해 강의자가진단도구를 개발하고, 도구의 타당화 및 관련 변인에 대한 분석을 실시하였다. 한 대학에서 강의를 담당했던 116명의 교ㆍ강사를 대상으로 8개 배경변인 이외에 23개의 자가진단 문항을 필요도와 이행도별로 응답하도록 했다. 다국면 라쉬모형을 적용한 문항 적합도 분석에 의해 이행도에 있어서 한 개를 제외하고 문항들이 모두 타당하다는 것을 알 수 있었다. 요인분석 결과는 문항을 구성하는 요인이 필요도와 이행도간에 차이가 발생한다는 것을 보여주었으며, 반응범주곡선을 이용하여 자가진단도구에서 5점 평정척도를 사용한 것이 적절하다는 것을 알 수 있었다. 내적일관성 신뢰도분석 결과는 비교적 높은 신뢰도를 보여주었지만, 필요도에 비해 이행도는 낮은 수준을 보였다. 한편 자가진단 결과는 필요하다고 인식하는 것보다 실제 이행하는 수준이 낮게 나타났고, 자가진단 수준에 영향을 미치는 관련 변인으로는 필요도에 있어서 교수의 소속대학과 연령 및 강좌규모가 있었으며, 이행도에 있어서는 교수의 성별이 발견되었다. This paper develops and validates the self-diagnosis instrument and analyzes the related variables for university instruction to provide professors with the opportunities of self-reflecting their own teaching. The instrument was administered to 116 instructors in a university and consists of 23 items for the two parts of "necessity" and "implementation" respectively. Infit statistics computed by FACETS, which is developed based on many-facet Rasch model, show that items are generally valid for the self-diagnosis of instruction except the item of "scolding students". Factor analysis finds that four factors of instruction process, student management, learning management, instruction method construct self-diagnosis of instruction for "necessity", while five factors of instruction method, instruction performance, instruction content, student management, learning management for "implementation". Also, the examination of response category curves provided by FACETS confirms the validity of 5 point rating scale for the instrument. Meanwhile, the results of administering the instrument reveal that instructors" gender, major, and age and the size of class can influence the level of instruction. As a result, the self-diagnosis instrument developed for university instruction can be a valid tool for evaluating instruction and improve the quality of university instruction, but further research is required for making the diagnosis of ‘implementation’ more objective.

比喩와 類推 및 心理를 활용한 子貢의 설득 커뮤니케이션 전략 검토 - 『論語』와 『史記』 「仲尼弟子列傳」을 중심으로

이연훈(Lee, Yon-Hun),김원중(Kim, Won-Joong) 동북아시아문화학회 2020 동북아 문화연구 Vol.1 No.64

Based on this study, Zigong’s persuasive communication strategies which use metaphor, simile, and inference are summarized into in the following manner. Confucius’s tendency of realpolitik implied in 『Lunyu』 influenced Zigong’s political-diplomatic skills, corresponding with 「Zhongni Dizi Liechuan」. Persuasive skills that Zigong showed can be pointed out as follows. First, conversations between Zigong and Confucius in 『Lunyu』 were mostly about political participation. In those incisive questions, persuasion to generalize and inference of future gave people trust and connected to high-level persuasive skills. Not only allusions in 『Shih Ching』 make readers understand the meaning of story by enhancing literary expressions and readers’ imagination, but also tendency of traditionalism becomes the norm of ethical and political assessment in the present. Second, Zigong became the main agent of persuasion and mentioned metaphor and simile as a communication skill in the conversations without Zigong and Confucius. A metaphor means the method which alludes the object to another object in common. Zigong emphasized the difference with Confucius by alluding Confucius to ‘the highest wall,’ ‘sun and moon,’ and ‘the heavens’. Also, there is an aspect of Confucianism with simultaneously respecting his teacher and showing his great realpolitik-ability. Third, diplomatic skills in 『Shiji』 「Zhongni Dizi Liechuan」 became the most effective persuasion by applying both logic and emotion. It looked into avoidable or charming factors, then stimulated the opponent with ‘antimony theory’ and ‘irony’. Additionally, emotions such as ‘favor,’ ‘fear,’ and ‘anger’ were applied at the right point, accompanying ‘inference’. It was possible to persuade someone with fine logic and ability to identify one’s mind as well as morality, Ren(仁). In other words, each circumstance that countries have, king’s competence to adapt, and logical fluency were essential conditions. Fourth, the biggest persuasive factor is in ‘improving(切磋琢磨).’ and ‘guess and hit(億則屢中)’. Someone who was eloquent but not prudent, was refined by Confucius’s education and Zigong’s effort. The ability of ‘guess and hit(億則屢中)’ emerged when having conversation with people and achieved saving Lu with diplomacy. As this study noted, Zigong’s persuasive communication strategy used rhetorical competence and psychology. Besides, it has advantage at speaking in today’s perspective.

우리나라 교육현장에 적합한 교사관심사 측정도구 타당화 연구

지은림(Chi, Eunlim),양명희(Yang, Myunghee),조윤정(Cho, Yoonjung) 한국교육평가학회 2009 교육평가연구 Vol.22 No.2

This paper aims to validate the measurement instrument of teacher concern which was developed in the U.S. since teacher concern is closely related with teachers' culture and environments. This study constructed the teacher concern scale based on Fuller's theory which assumes that teacher concern consists of three factors of ‘impact’, ‘self’, and ‘task’. George's TCQ (Teacher Concern Questionnaires), which included 44 five-point scale items, was administered to 1,077 secondary teachers. To validate the measurement instrument, the analysis of item fit and item response categories was done by many-faceted Rasch model. The analysis identified 9 misfitting items, and the misfitting items revealed that Korean teacher concern reflects different aspects from Americans. After deleting misfitting items, the interval of item scale was improved and Cronbach-α was also increased. In addition, Mplus analysis for confirmatory factor analysis was applied to explore the relationships between latent trait factors of teacher concern. Among three factors, ‘impact’ was found as the first factor, and ‘self’ was the second factor. In terms of the level of teacher concern, the average score of ‘self’ was highest, and that of ‘task’ was lowest. All of these results suggest that the quality of teacher concern measurement instrument can be enhanced by considering the characteristics of Korean schools. 교사관심사는 수업과 관련하여 교사들이 지각하는 문제와 요구, 어려움으로 정의되는 바 교사교육 분야에서 오랫동안 연구되어온 교사의 심리적 특성이다. 본 연구는 미국의 교육상황에서 개발된 ‘교사관심사’ 측정도구가 우리나라 교사들에게도 적용가능한지를 살펴보기 위해 측정학적 측면들을 검토하였다. 다국면라쉬모형을 적용하여 문항의 적합성과 문항반응범주의 적절성을 검토하였는데, 부적합 문항으로 나타난 9개 문항들은 학급당 학생 수나 학교업무 등과 관련하여 우리나라 교육현실에서 교사의 자율권이 낮은 문항이 교사관심사가 되기 어렵다는 것을 보여주었으며, 동료교사에 관한 관심도 미국과는 다른 양상을 보였다. 부적합한 문항을 제외한 문항으로 구성한 질문지를 이용하여 신뢰도 검토와 문항반응범주를 재확인하였는데, 질문지가 측정학적으로 개선되었다는 것을 알 수 있었다. 또한 확인적 요인분석을 통해 요인의 특성에 대한 탐색을 실시하였고, 개발된 질문지 결과에 따른 교사들의 관심사 수준을 파악해보았다. 결론적으로 외국에서 사용하던 교사관심사 도구를 그대로 사용하기 보다는 우리나라 실정에 맞게 수정, 보완하는 것이 교사관심사에 대한 정확한 진단을 위해 필요하다는 것을 알 수 있었다.