컴퓨터 적응검사에 대한 노출문항 효과 = The effect of compromised items for computerized adaptive testing

국문 초록 (Abstract)

본 연구의 목적은 컴퓨터 적응검사에서 사전에 노출된 문항이 수험자의 능력 추정에 어떤 영향을 주는 지 탐색하는 것이다. 노출된 문항이 수험자의 능력을 과대 추정하겠지만, 그 정도에 대한 객관적인 연구가 미미하다. 본 연구에서는 컴퓨터 적응검사 시뮬레이션을 통해 노출문항의 영향력을 구체적으로 확인하고자 하였다. 문항선택 방법으로 최대문항정보선택법(MI), a-다층화 방법(AS)과 더불어 문항 노출을 조정하는 심슨-해터 방법(MISH, ASSH)을 사용하였고 노출된 문항 오답의 정답 전환률을 0%~100%로 설정하여 그에 따른 측정의 정확성과 문제은행의 활용도를 확인하였다. 그 결과 노출된 문항이 존재하지 않을 때, MI의 측정의 정확성이 가장 좋았지만 문제은행 활용도는 낮았다. 그러나 전환률이 증가할수록 ASSH, AS, MISH 순으로 측정의 정확성이 좋았다. 일단 노출된 문항이 수험자의 능력 추정에 영향을 주기 시작하면 측정의 정확성을 재는 bias와 MSE가 급격히 높아지고 문제은행의 활용률의 척도인 X²도 높아져 문제은행의 활용이 경직됨을 알 수 있었다. 능력추정 정확성과 문제은행 활용률을 고려할 때, 고부담 시험의 경우 ASSH, AS가 MI, MISH보다는 검사 보안을 위해서 더 적절한 문항선택 방법일 수 있다.

다국어 초록 (Multilingual Abstract)

The purpose of this research is to explore the effect of artificially compromised items for computerized adaptive testing. It is obvious for the compromised items to overestimate examinee's ability, but there are not many empirical researches to support our guess. For this study, I performed the simulation study to explore the effect of compromised items with maximum information item selection method(MI), a-stratified multi stage method(AS), and the Sympson-Hetter method incorporated with these two item selection methods(MISH, ASSH). The change rates of compromised items from a wrong response to a right response have been set as 0%~100%. Along with the change rates, bias and MSE for measurement precision and X² for item bank's usages were compared. When an item bank did not have compromised items, MI showed the lowest bias and MSE, but the highest X². However, as the change rates have been increased, MI's bias and MSE dramatically increased. Also biases and MSE's of other algorithms dramatically increased, but ASSH and AS showed lower bias, MSE, X² than MI and MISH did. Thus, for high stake tests, ASSH and AS might be more appropriate algorithms than MI and MISH to protect test security.

목차 (Table of Contents)

• 국문초록
• Ⅰ. 서론
• Ⅱ. 이론적 배경
• Ⅲ. 연구방법
• Ⅳ. 연구결과
• Ⅴ. 결론 및 논의사항
• 참고문헌

참고문헌 (Reference)

1 채선희, "컴퓨터화 개별적응검사와 지필검사에 있어서 문항모수치의 동등성 검증" 8 (8): 145-160, 1995

2 반재천, "문제은행간 비동등성이 능력모수 추정치에 미치는 영향" 한국교육평가학회 16 (16): 79-92, 2003

3 부재율, "대학교 총괄평가를 이용한 지필검사와 컴퓨터활용검사의 비교성 검증" 한국교육평가학회 21 (21): 175-199, 2008

4 Chang, H., "a-Stratified Multistage Computerized Adaptive Testing" 23 : 211-222, 1999

5 Lord, F. M., "Tailored testing, an approximation of stochastic approximation" 66 : 707-711, 1971

6 Veerkamp, W. J. J, "Some new item selection criteria for adaptive testing" 22 : 203-226, 1997

7 van der Linden, W. J., "Some alternatives to Sympson-Hetter item exposure control in computerized adaptive testing" 28 : 249-265, 2003

8 Yi, Q., "Severity of organized item theft in computerized adaptive testing: A simulation study" 32 : 543-558, 2008

9 Zubrzycki, J., "Report: preventing, detecting, and responding to cheating"

10 Kingsbury, G. G., "Procedure for selecting items for computerized adaptive tests" 2 : 359-375, 1989

11 시기자, "PPT, CFT, CAT에서 검사매체와 검사시행 모형에 따른 피험자 응답 적합도의 비교" 한국교육평가학회 19 (19): 65-87, 2006

12 시기자, "PPT, CFT, CAT에서 검사매체와 검사시행 모형에 따른 피험자 능력모수의 비교" 한국교육평가학회 17 (17): 53-78, 2004

13 Drasgow, F., "Optimal detection of certain forms of inappropriate test scores" 10 : 59-67, 1986

14 Davey, T., "New algorithms for item selection and exposure control with Computerized Adaptive Testing" 1995

15 McBride, J. R., "New Horizons in testing: Latent trait test theory and computerized adative testing" AcademicPress 223-236, 1983

16 Levine, M., "Measuring the appropriateness of multiple-choice test scores" 4 : 269-289, 1979

17 Leung, C-K., "Item selection in computerized adaptive testing : Improving the alpha-stratified design with the Sympson-Hetter algorithm" 26 : 376-392, 2002

18 Hau, K., "Item Selection in Computerized Adaptive Testing: Should More Discriminating Items be Used First?" 38 : 249-266, 2001

19 Chang, H., "Hypergeometric family and test overlap rates in computerized adaptive testing" 67 : 387-398, 2002

20 McLeod, L. D., "Detecting item memorization in the CAT environment" 23 : 147-160, 1999

21 Sympson, J. B., "Controlling item-exposure rates in computerized adaptive testing" Navey Personnel Research and Development Center 973-977, 1985

22 Stocking, M, L., "Controlling item exposure conditional on ability in in computerized adaptive testing" 23 : 57-75, 1998

23 Davey, T., "Computer based testing: Building the foundation for future assessments" LawrenceErlbaumAssociates 165-191, 2002

24 Celis, W., "Computer Admissions Test Found to Be Ripe for Abuse"

25 Hetter, R. D., "CAT: From inquiry to operation" American Psychological Association 1997

26 van der Linden, W. J., "Bayesian item-selection criteria for adaptive testing" 63 : 201-216, 1998

27 Lord, F. M., "Applications of item response theory to practical testing problems" Lawrence Erlbaum Associates 1980

28 Segall, D. O., "An item response model for characterizing test compromise" 27 : 163-179, 2002

29 Bayroff, A. G., "An exploratory study of branching test" Army Behavioral Science Research Laboratory 1967

30 Weiss, D. J., "Ability measurement: conventional or adaptive?" University of Minnesota, Department of Psychology 1973

31 Segall, D. O., "A sharing item response theory model for computerized adaptive testing" 29 : 439-460, 2004

32 Zhang, J, "A sequential procedure for detecting compromised items in the item pool of a CAT system" 38 : 87-104, 2014

33 Stocking, M, L., "A new method of controlling item exposure in computerized adaptive testing" EducationalTestingService 1995

34 Chang, H., "A global information approach to computerized adaptive testing" 20 : 213-229, 1996

35 Chang, S. W., "A comparative study of item exposure control methods in computerized adaptive testing" 40 : 71-103, 2003

36 McLeod, L. D., "A Bayesian method for the detection of item preknowledge in computerized adaptive testing" 27 : 121-137, 2003