일반화가능도 이론을 적용한 단위검사 구성 검사점수의 신뢰도 추정 = Estimating Reliability of Test Scores Composed of Testlets using Generalizability Theory Approaches

국문 초록 (Abstract)

  단위검사로 구성된 검사 점수의 신뢰도를 추정할 때 문항점수를 이용한 Cronbach α 방법으로 신뢰도를 추정하면 검사 점수의 신뢰도를 과대 추정하게 된다. 이 논문은 국내측정 자료를 대상으로 아직 실증연구가 없었던 단위검사로 구성된 검사점수의 신뢰도를 일반화가능도 이론을 적용하여 추정하고, Cronbach α로 산출한 신뢰도의 과대 추정 규모를 확인, 적절한 신뢰도 추정 방법 모색을 목적으로 한다. 이 연구는 학업성취도 자료가 아닌 적성검사의 하위검사인 사고유연성 검사 자료를 수집하여 분석하였다는 측면에서 외국의 선행 연구들과 차별성을 가진다. 이 연구에서는 전통적인 신뢰도 추정방법인 Cronbach α와 일반화가능도 이론을 적용한 신뢰도 추정방법 등 총 5가지 방법을 적용하여 신뢰도를 추정하였으며, 이를 통해 Cronbach α의 과대추정 규모가 약 0.1수준이고, 이러한 과대추정은 단위검사 내 문항의 상관이 단위검사 간 문항의 상관보다 높기 때문임을 확인할 수 있었다. 목표로 하는 신뢰도 확보를 위해서는 검사의 문항 수가 고정된 상태에서 단위검사 내의 문항 수를 늘리는 것 보다 단위검사의 수를 증가시키는 것이 효율적이라는 결과를 도출하였다.

다국어 초록 (Multilingual Abstract)

  When Cronbach α is used in estimating the reliability of test scores composed of testlets, it may overestimate the reliability of test scores. This study was designed to address issues of exploring several procedures of estimating the reliability of test scores composed of testlets and examining the degree of overestimation when using Cronbach α instead of using proper methods. Generalizability theory approaches were adopted to investigate these research purposes and questions. This study is the first empirical study dealing with the testlet concept under generalizability theory frameworks in Korea. Another academic contribution of this study can be found in using aptitude tests rather than using achievement tests which were most frequently used in previous studies conducted in the united states and other countries. Based upon major findings of this study, following conclusions can be made. The overestimation size of using Cronbach α instead of using proper methods is about 0.1 which is slightly bigger than those reported in similar previous studies. The overestimation seems to be mainly caused by the fact that the average of within-testlet inter-item correlations is higher than that of between-testlet inter-item correlations. To have a targeted reliability, it could be more efficient way to increase the number of testlets than to increase the number of items within testlets if the total number of items in a test is fixed.

목차 (Table of Contents)

• Ⅰ. 서론
  Ⅱ. 단위검사와 일반화가능도 이론 모형
  Ⅲ. 연구 방법
  Ⅳ. 연구 결과
  Ⅴ. 결론 및 논의
  참고문헌
  저자소개
  〈ABSTRACT〉
  

참고문헌 (Reference)

1 "중학생용 적성검사 개발 1차년도 연구 보고서" 서울: 한국고용정보원 2005

2 "언어영역 하부검사의 지역의존성과 모수추정모형의 탐색" 18 (18): 102-120, 2005

3 "단위검사 개념의 적용 -일반화가능도 이론을 중심으로-" 16 (16): 53-70, 2003

4 "교육평가 용어사전" 서울: 학지사 2004

5 "Using a new statistical model for testlets to score TOEFL" 37 : 203-220, 2000

6 "Trace lines for testlets A use of multiple- categorical models Journal of Educational Measurement" 247-260, 1989

7 "The use of application test a novel type of problem-solving exercise as a tool of teaching and assessment of competence in medical biology" 73-82, 1987

8 "The relative appropriateness of eight measurement models for analyzing scores from tests composed of testlet" 61 : 971-988, 2001

9 "The influence of several factors on reliability for complex reading comprehension tests" 39 : 149-164, 2002

10 "The dependability of behavioral measurements Theory of generalizability for scores and profiles" 1972

11 "The Nature of Human Intelligence" McGraw-Hill Book Company 1967

12 "Signal/Noise Ratios for Domain-Referenced Tests" 609-25, 1977b

13 "Score reliability of a test composed of passage-based testlets" 2 (2): 1-29, 2003

14 "Scaling performance assessments:Strategies for managing local item dependence" 30 : 1993

15 "Precision and differential item functioning on a testlet-based test:The 1991 Law School Admissions Test as an example" 8 : 157-186, 1995

16 "On the reliability of testlet-based tests Journal of Educational Measurement" 1991

17 "Manual for urGENOVA Version 2.1" Iowa Testing Programs. Iowa city, IA (49) : 2001b

18 "Item Response Theory for Psychologist" NJ: Lawrence Erlbaum Associates 2000

19 "Incorporating the testlet concept in test score analyses" 19 : 9-15, 2000

20 "How is reliability related to the quality of test scores? What is the effect of local dependence on reliability?" 15 (15): 22-29, 1996

21 "Generalizability theory" New York: Springer 2001a

22 "Estimating reliability under a generalizability theory model for test scores composed of testlets" 12 : 237-255, 1999

23 "Elements of generalizability theory" American College Testing 1992

24 "Development of a taxanomy of human performance A feasibility study of ability dimensions for classifying human task JSAS Catalog of Selected document in Psychology" 25-26, 1973

25 "An Index of Dependability for Mastery Tests Journal of Educational Measurement" 277-89, 1977a

26 "American Council on Education" 1951

27 "A comparison of methods of estimating conditional standard errors of measurement for testlet-based test scores using simulation techniques" 37 : 91-112, 2000

28 "A case for testlets Journal of Educational Measurement" 1987