Do Subscores from a Summative English Listening Test Have Added Value? = 영어 듣기시험의 세부 영역 점수가 갖는 신뢰도와 타당도 연구

국문 초록 (Abstract)

본 연구에서는 총합평가의 특성을 갖는 영어시험으로부터 진단적 성격을 갖는 유용한 세부영역 점수를 score augmentation을 통하여 도출할 수 있는지를 탐색하였다. 실제 중고등학교에서 사용되는 영어 시험에서 세부 영역 점수의 측정학적 특성을 조사하기 위하여, 네 개의 세부영역을 갖는 100개의 문항으로 구성된 대규모 영어 듣기시험으로부터 동일한 세부영역을 갖는 세 가지 서로 다른 크기의 시험을 (문항 개수 20, 32, 40)을 각각 5, 8, 10 개의 세부영역 문항을 이용하여 구성하였다. 측정학적 분석에 의하면 이 세부영역 점수의 신뢰도는 일반적인 시험에서 요구되는 정도에 훨씬 못 미치는 것이었으나, 다른 세부영역 점수에 대한 정보를 바탕으로 한 score augmentation 과정을 거친 후에는 상당한 정도의 신뢰도 향상을 보였다. 그러나 신뢰도가 향상된 점수의 대부분은 각 수험자 집단의 평균점수로 회귀하는 현상을 보여, 조정된 세부영역 점수의 타당도를 의심케하였다. 이 후 이 연구에서는 이러한 회귀현상에 대한 원인과 이 연구의 결과가 영어 시험 제작에 갖는 의미를 논의하였다.

다국어 초록 (Multilingual Abstract)

This study was to explore the potential for diagnostic subsocres from a summative English proficiency test. From an English listening comprehension test that consisted of 100 items with four subdomains, three different sizes of test (20, 32, and 40 item tests) with the same four subdomains were constructed to simulate subscore properties of tests in practice. The number of items for the subdomains of the three simulated tests were 5, 8, 10, respectively. Statistical analyses showed that the raw subscroes had unacceptable reliabilities but the subsequent score augmentation technique increased the reliabilities remarkably. However, the study also found out that such gains were obtained by heavily regressing individual subscores toward the profiles of the group means, resulting in very similar subsocres, which in turn questioned the validity of such augmented subscores. The study searched for the causes of the problem and discussed implications for constructing a test that can provide diagnostic information.

목차 (Table of Contents)

• Abstract
• Ⅰ. Introduction
• Ⅱ. Literature Review
• Ⅲ. Methods
• Ⅳ. Results
• Ⅴ. Discussion and Implications
• References

참고문헌 (Reference)

1 Haberman, J. S., "When can subscores have value?" ETS 2005

2 Sinharay, S., "When can subscores be expected to have added value? Results from operational and simulated data" ETS 2010

3 Vevea, J. L., "User's guide for Augment v.2: Empirical Bayes subscore augmentation software" University of North Carolina, L.L. Thurstone Psychometric Laboratory 2002

4 Reise, S. P., "The role of the bifactor model in resolving dimensionality issues in health outcomes measures" 16 : 19-31, 2007

5 Monaghan, W., "The facts about subscores" ETS 2006

6 Wainer, H., "Test Scoring" Lawrence Erlbaum 343-387, 2001

7 Sinharay, S., "Subscores based on classical test theory : To report or not to report" 26 (26): 21-28, 2007

8 Goodman, D. P., "Student test score reports and interpretive guides : Review of current practices and suggestions for future research" 17 : 145-220, 2004

9 American Educational Research Association, "Standards for educational and psychological testing"

10 Haberman, J. S., "Reporting subsocres: A survey" ETS 2008

11 Haberman, J. S., "Reporting subscores for institutions" 62 : 79-95, 2009

12 Stone, C. A., "Providing subscale scores for diagnostic information : A case study when the test is essentially unidimensional" 23 : 63-86, 2010

13 Harris, D. J., "Methods of examining the usefulness of subscores" 1991

14 Cai, L., "IRTPRO for Windows [Computer software]" Scientific Software International 2011

15 Paek, I., "IRTPRO 2. 1 for Windows(Item Response Theory for Patient-Reported Outcomes)" 37 (37): 242-252, 2013

16 Kelly, T. L., "Fundamentals of statistics" Harvard University Press 1947

17 Gibbons, R. D., "Full-information item bi-factor analysis" 57 : 423-436, 1992

18 Reise, S. P., "Bifactor models and rotations : Exploring the extent to which multidimensional data yield univocal scale scores" 92 (92): 544-559, 2010

19 Edwards, M. C., "An empirical Bayes approach to subscore augmentation: How much strength can we borrow?" 31 (31): 241-259, 2006

20 Sinharay, S., "An NCME instructional module on subscores" 30 (30): 29-40, 2011

21 Skorupski, W. P., "A comparison of approaches for improving the reliability of objective level scores" 70 (70): 357-375, 2010

22 Yen, W. M., "A Bayesian/IRT index of objective performance for tests with mixed-item types" 1997

23 Yen, W. M., "A Bayesian/IRT index of objective performance" 1987