학문 목적 한국어 쓰기 능력에 대한 분석적 평가의 일반화가능도 검증 = Analysis of Error Sources and Estimation of Reliability in a Analytic Evaluation on Writing Ability for Academic Purpose Korean by Applying Generalizability Theory.

다국어 초록 (Multilingual Abstract)

The primary aim of this study is: 1) to determine error sources and the effects of each error source in an analytic evaluation on APK(Academic Purpose Korean) writing ability; 2) to estimate the generalizability coefficient; and 3) to suggest optimal measuring conditions for an analytic evaluation on APK writing ability based on the generalizability theory.
For the design of the G study(Generalizability Study), a three-facet crossed model(p x r x i) was used with two academic writing task(a test with 2 hour limit and a report with 1 week term), 9 raters, and 10 analytic scoring items. The D study(Decision Study) design was used in an identical manner as the G study. The result of the G study for estimating error sources in the analytic evaluation on APK writing ability indicates that error sources for raters have a more significant impact than error sources related to analytic scoring items associated with the score of the writing test.
As for the generalizability coefficients in the D study, increasing the number of the raters had more significant impact than increasing the number of assessment items in the analytic evaluation on APK writing ability. However even the ratio in the error dispersion of the raters is not small, the ratio in the error dispersion of the items is 0. It means that these 10 analytic scoring items guided a coherent response from 9 raters. Therefore, it seems useful to train the raters with these 10 analytic scoring items.

참고문헌 (Reference)

1 강애남, "학생들의 동료평가를 활용한 수행평가 결과의 일반화가능도 분석" 한국교육평가학회 19 (19): 107-121, 2006

2 김정숙, "학문적 목적의 한국어 교육과정 설계를 위한 기초 연구" 국제한국어교육학회 11 (11): 1-20, 2000

3 김성숙, "학문 목적 한국어 쓰기 능력 평가 척도 개발과 타당성 검증" 연세대학교 2011

4 Brennan, R. L., "일반화가능도이론" 서울: 연세대학교 출판부 1983

5 조재윤, "일반화가능도 이론을 이용한 쓰기 평가의 오차원 분석 및 신뢰도 추정 연구" 한국어교육학회 (128) : 325-358, 2009

6 김성숙, "일반화가능도 이론" 서울: 교육과학사 2001

7 김성규, "외국인 및 재외국민을 위한 대학국어 이수 방법 및 교육 내용 개선을 위한 기초적 연구" 서울대학교 기초교육원 사업보고서, 서울대학교 기초교육원 2005

8 석주연, "외국인 대학 신입생의 한국어 능력 실태 연구" 가톨릭대학교 인문과학연구소 (11) : 65-82, 2006

9 성태제, "문항반응이론의 이해와 적용" 서울: 교육과학사 16-, 2001

10 김성숙, "내·외국인 학생 간의 작문 수정 결과 비교" 국어국문학회 (153) : 377-416, 2009

11 Cronbach, L. J., "The Generalizability of Behavioral Measurement: The Theory of generalizability for scores and profiles" New York: John Wiley 1972

12 Baker,F.B, "The Basic of Item Response Theory" Portsmouth, N. H.: Heinemann 1985

13 Shavelson, R., "Sampling variability of performance assessment" 30 (30): 215-235, 1993

14 Shavelson, R, "Measurement of teaching, In Handbook of research on teaching(3rd ed.)" New York:Macmillan 50-91, 1986

15 Crick, J. E, "Mannual for GENOVA: A Generalized analysis of variation system" ACT Program, Iowa city, IA. 43 : 1983

16 Brennan, R., "Elements of Generalizability Theory(2nd)" Iowa city, IA: The American College Testing Program 1983