The relationship between classical item characteristics and item response time on computer-based testing

다국어 초록 (Multilingual Abstract)

Purpose: This study investigated the relationship between the item response time (iRT) and classic item analysis indicators obtained from computer-based test (CBT) results and deduce students’ problem-solving behavior using the relationship.
Methods: We retrospectively analyzed the results of the Comprehensive Basic Medical Sciences Examination conducted for 5 years by a CBT system in Dankook University College of Medicine. iRT is defined as the time spent to answer the question. The discrimination index and the difficulty level were used to analyze the items using classical test theory (CTT). The relationship of iRT and the CTT were investigated using a correlation analysis. An analysis of variance was performed to identify the difference between iRT and difficulty level. A regression analysis was conducted to examine the effect of the difficulty index and discrimination index on iRT.
Results: iRT increases with increasing difficulty index, and iRT tends to decrease with increasing discrimination index. The students’ effort is increased when they solve difficult items but reduced when they are confronted with items with a high discrimination. The students’ test effort represented by iRT was properly maintained when the items have a ‘desirable’ difficulty and a ‘good’ discrimination.
Conclusion: The results of our study show that an adequate degree of item difficulty and discrimination is required to increase students’ motivation. It might be inferred that with the combination of CTT and iRT, we can gain insights about the quality of the examination and test behaviors of the students, which can provide us with more powerful tools to improve them.

참고문헌 (Reference)

1 임현선, "문항반응이론과 고전검사이론을 이용한 진료수행시험의 문항 분석" 한국의학교육학회 19 (19): 185-196, 2007

2 University of Washington, "Understanding item analyses"

3 Wise SL, "Response time effort : a new measure of examinee motivation in computer-based tests" 18 (18): 163-183, 2005

4 Kingsbury GG, "Procedures for using response latencies to identify unusual test performance in computerized adaptive tests" 1993

5 Huh S, "Preparing the implementation of computerized adaptive testing for high-stakes examinations" 5 : 1-, 2008

6 Bovaird JAI, "New applications in testing: using response time to increase the construct validity of a latent trait estimate"

7 Lee YH, "Introduction to an open source internet-based testing program for medical student examinations" 6 : 4-, 2009

8 Schoening A, "Interpreting exam performance: what do those stats mean anyway?"

9 Yang CL, "Examining item difficulty and response time on perceptual ability test items" 3 (3): 282-299, 2002

10 Halkitsis PN, "Estimating testing time: the effects of item characteristics on response latency" 1996

11 Son CG, "Curriculum & educational evaluation" Teayong 2012

12 Wolf LF, "Consequence of performance, test, motivation, and mentally taxing items" 8 (8): 341-351, 1995

13 Bergstrom BA, "Computerized adaptive testing exploring examinee response time using hierarchical linear modeling" 1994

14 Schnipke DL, "Computer-Based Testing: Building the Foundation for Future Assessments" Lawrence Erlbaum Associates 244-, 2002

15 Wise SL, "An investigation of the differential effort received by items on a low-stakes computer-based test" 19 (19): 95-114, 2006

16 Wise SL, "An investigation of the differential effort received by items on a low-stakes computer-based test" 19 (19): 95-114, 2006