다변량 일반화가능도 이론에 의한 말하기 시험의 타당도와 신뢰도에 관한 연구 = An investigation into the dependability of ratings in a German speaking test using the multivariate generalizability theory

다국어 초록 (Multilingual Abstract)

Even though performance-based language tests are often multivariate by design, previous validation studies have analyzed the dependability of ratings of such tests within univariate analytic frameworks. In an attempt to address this limitation, the present study investigated the dependability of judgements made based on ratings of a German speaking test using a multivariate generalizabilitiy theory (MGT) analysis. Data obtained from 88 students were analyzed in G- and D-studies of a two-facet crossed design. A D-study with two raters and two tasks supported the high dependability of the ratings and the placement decisions made at the pre-determined cut scores. An optimization analysis of the measurement procedure suggested that the desired level of dependability of ratings has already been achieved by the use of two raters and two tasks. The MGT analysis also generated other useful information about the convergent/ discriminant validity of the five subscales. Specifically, the universe score variance-covariance matrix obtained from the D-study showed that the underlying subscales were interrelated but distinct. Furthermore, the analysis of effective weights of the scales revealed that the Grammar subscale played the dominant role in the composite universe score variance.

목차 (Table of Contents)

• Abstract
• Ⅰ. 서론
• Ⅱ. 이론적 배경
• Ⅲ. 연구방법
• Ⅳ. 연구결과
• Ⅴ. 결론 및 제언
• 참고문헌

참고문헌 (Reference)

1 이영식, "최근 언어평가의 연구와 이론적 배경" 52 (52): 221-246, 1997

2 이영식, "영어작문평가의 채점신뢰도에 대한 분석" 53 (53): 179-200, 1998

3 Brennan, R. L, "mGENOVA, Version 2.0"

4 Lynch, B, "Using G-theory and many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants" 15 (15): 158-180, 1998

5 Weigle, S, "Using FACETS to model rater training effects" 15 : 263-287, 1998

6 Brown, J. D, "The relative importance of persons, items, subtests and languages to TOEFL test variance" 16 (16): 217-238, 1999

7 Brown, A, "The effect of rater variables in the development of an occupation-specific language performance test" 12 : 1-15, 1995

8 McNamara, T. F, "The effect of interlocutor and assessment mode variables in overseas assessments of speaking skills in occupational settings" 14 : 140-156, 1997

9 Cronbach, L. J, "The dependability of behavioral measurements: Theory of generalizability for scores and profiles" New York: Wiley 1972

10 최인철, "Test of Oral Proficiency(TOP)의 개발 연구" 34 (34): 245-286, 1998

11 Stansfield, C. W, "Research of the comparability of the oral proficiency interview and the stimulated oral proficiency interview" 20 : 347-364, 1992

12 Nussbaum, A, "Multivariate generalizability theory in educational measurement: An empirical study" 8 (8): 219-230, 1984

13 Webb, N. M, "Multivariate generalizability theory (In Generalizability theory: Inferences and practical applications)" San Francisco, CA: Jossey-Bass 67-81, 1983

14 Webb, N. M, "Multivariate generalizability of general educational development ratings" 18 (18): 13-22, 1981

15 Bachman, L, "Investigating variability in tasks and rater judgments in a performance test of foreign language speaking" 12 (12): 238-257, 1995

16 Shavelson, R. J, "Generalizability theory: 1973-1980" 34 : 133-166, 1981

17 Bachman, L., "Generalizability theory (In Encyclopedia of language and education. Volume 7: Language testing assessment)" Dordrecht: Kluwer 255-262, 1997

18 Brennan, R. L, "Generalizability theory" New York: Springer-Verlang 2001

19 Shavelson, R. J, "Generalizability theory" 6 : 922-932, 1989

20 Crick, J. E, "GENOVA: A generalized analysis of variance system" Dorchester, MA: Computer Facilities, University of Massachusetts at Boston 1982

21 Choi, Y. H, "GENOVA and FACETS analysis of English writing test: Tasks, raters, and scales" 56 (56): 125-142, 2001

22 Bachman, L. F, "Fundamental considerations in language testing" Oxford: Oxford University Press 1990

23 Marcoulides, G. A, "Estimating variance components in generalizability theory: The covariance structure analysis approach" 3 (3): 290-299, 1996

24 Brennan, R. L., "Elements of generalizability theory" Iowa City, IA: The American College Testing Program 1992

25 Bachman, L, "EAP language ability assessment system: Interim report" Los Angeles: University of California, Los Angeles, Department of TESL and Applied Linguistics 1992

26 Bolus, R. E, "An introduction to generalizability theory in second language research" 32 : 245-258, 1982

27 Bonk, W. J, "A many-facet Rasch analysis of second language group oral discussion task" 20 : 89-110, 2003

28 Brown, J. D, "A categorical instrument for scoring second language writing skills" 34 (34): 21-22, 1984