문항반응이론을 적용한 단위검사 구성 검사점수의 신뢰도 추정 = Estimating Reliability of Test Scores Composed of Testlets Using Item Response Theory Approaches

국문 초록 (Abstract)

  단위검사로 구성된 검사 점수의 신뢰도를 산출하기 위해 전통적인 Cronbach α 방법을 적용하게 되면 검사 점수의 신뢰도는 과대 추정된다. 이 논문은 단위검사로 구성된 검사점수의 신뢰도를 문항반응 이론의 다양한 문항반응모형을 적용하여 추정하고, Cronbach α로 산출한 신뢰도의 과대 추정 규모를 확인하여 문항반응이론 이용 시 적절한 신뢰도 추정 방법을 모색하려는 목적을 갖는다. 이 연구는 문항반응이론의 관점에서 신뢰도를 제고하고, 단위검사문항반응모형을 적용하여 실제 자료를 분석한 연구로는 국내에서 처음 시도되었다는 측면에서 선행연구들과 차별성을 갖는다. 전통적인 신뢰도 추정방법인 Cronbach α와 문항반응이론을 적용한 신뢰도 추정 방법 등 총 5가지 방법을 적용하여 신뢰도 계수를 산출하였고, 이러한 분석을 통해 Cronbach α 방법이 신뢰도를 약 0.07수준으로 과대추정함을 확인할 수 있었다. 이 연구에서는 문항을 단위검사 수준으로 합산하여 단위검사 점수를 생성하고 다분문 항반응모형을 적용한 신뢰도 추정방법과 문항 및 단위검사 효과를 동시에 고려하는 단위검사문항반응 모형을 적용한 신뢰도 추정방법은 그 차이가 약 0.01수준으로 작았으며, 이는 다분문항반응모형을 적용할 때 정보의 손실 정도가 비교적 크지 않은 것으로 해석될 수 있다.

다국어 초록 (Multilingual Abstract)

  When Cronbach α is used in estimating the reliability of test scores composed of testlets, it may overestimate the reliability of test scores. This study was designed to address issues of exploring several procedures of estimating the reliability of test scores composed of testlets and examining the degree of overestimation when using Cronbach α instead of using proper methods. Item response theory approaches were adopted to investigate these research purposes and questions. This study seems the first empirical study dealing with the testlet concept under item response theory frameworks in Korea. Another academic contribution of this study can be found in using testlet item response theory model that considers items and testlets at the same time by adding one more parameter of testlet effect in the item response model. Based upon major findings of this study, following conclusions can be made. The overestimation size of using Cronbach α instead of using proper methods was about 0.07 which seemed moderate size comparing to those reported in similar previous studies. In addition, the difference between reliability coefficients produced by both polytomous item response model and testlet item response model was about 0.01, which implies the degree of information loss in this context does not seem too serious.

목차 (Table of Contents)

• Ⅰ. 서론
  Ⅱ. 이론적 배경
  Ⅲ. 연구 방법
  IV. 연구 결과
  V. 결론 및 논의
  참고문헌
  저자소개
  〈ABSTRACT〉
  

참고문헌 (Reference)

1 "중학생용 적성검사 개발 1차년도 연구 보고서" 서울: 한국고용정보원 2005

2 "일반화가능도이론을 적용한 단위검사 구성 검사 점수의 신뢰도 추정" 20 (20): 119-139, 2007

3 "언어영역 하부검사의 지역의존성과 모수추정모형의 탐색" 18 (18): 102-120, 2005

4 "단위검사 개념의 적용 -일반화가능도이론을 중심으로-" 16 (16): 53-70, 2003

5 "Using a new statistical model for testlets to score TOEFL" 37 : 203220-, 2000

6 "User's guide for SCORERIGHT (ver. 2.0): A computer program for scoring tests built of testlets" 2001

7 "Toward a psychometrics for testlets Journal of Educational Measurement" 1990

8 "The use of application test a novel type of problem-solving exercise as a tool of teaching and assessment of competence in medical biology" 73-82, 1987

9 "The relative appropriateness of eight measurement models for analyzing scores from tests composed of testle" 61 : 971-988, 2001

10 "The influence of several factors on reliability for complex reading comprehension tests" 39 : 149-164, 2002

11 "The Nature of Human Intelligence" McGraw-Hill Book Company 1967

12 "Testlet response theory. An analog for the 3-PL useful in testlet-based adaptive testing" Boston, MA: Kliwer-Nijhoff 245-270, 2000

13 "Technical Guidelines For Assessing Computerized Adaptive Tests Journal of Educational Measurement" 347-360, 1984

14 "Strategies for managing local item dependence Journal of Educational Measurement" 1993187213

15 "Statistical theories of mental test scores" & (&): 1968

16 "Some latent trait models and their use in inferring an examinee's ability Statistical Theories of Mental Test Scores" 1968

17 "Precision and differential item functioning on a testlet-based test The 1991 Law School Admissions Test as an example" 8 : 157186-, 1995

18 "On the reliability of testlet-based tests Journal of Educational Measurement" 1991

19 "July 11 in Sireci On the reliability of testlet-based tests Journal of Educational Measurement" 19901991

20 "Item analysis and test scoring with binary logistic models" 1990

21 "Item Response Theory" Westport, CT: ACE 2006

22 "Introduction to Measurement Theory" Brooks/Cole 1979

23 "Incorporating the testlet concept in test score analyses" 19 : 9-15, 2000

24 "How is reliability related to the quality of test scores? What is the effect of local dependence on reliability?" 15 (15): 2229-, 1996

25 "Estimation of latent ability using a response pattern of graded scores" 1969

26 "Estimating reliability under a generalizability theory model for test scores composed of testlets" 12 : 237-255, 1999

27 "Estimating item parameters and latent ability when responsese are scored in two or more norminal categories" 37 : 29-53, 1972

28 "Development of a taxanomy of human performance A feasibility study of ability dimensions for classifying human task JSAS Catalog of Selected document in Psychology" 25-26, 1973

29 "Application of the Bi-Factor Multidimensional Item Response Theory Model to Testlet-Based Tests" 43 (43): 145-168, 2006

30 "Application of an EM algorithm" 159-176, 1992

31 "American Council on Education" 1951

32 "A comparison of methods of estimating conditional standard errors of measurement for testlet-based test scores using simulation techniques" 37 : 91-112, 2000

33 "A case for testlets Journal of Educational Measurement" 1987185201

34 "A Rasch model for partial credit scoring" 47 : 149-174, 1982

35 "A General Bayesian Model for Testlets: Theory and Applications" 26 : 109-128, 2002

36 "A Bayesian Random Effects Model For Testlets" 64 : 153-168, 1999