A Comparison of Kernel and Traditional Equating Methods under the Non-Equivalent Groups with Anchor Test Design

국문 초록 (Abstract)

이 연구는 비동등집단 가교문항 설계(이하 NEAT)의 조건에 따른 커넬 동등화 방법과 전통적 동등화 방법의 결과를 비교하였다. 특히, NEAT 설계에서 동등화 결과에 영향을 줄 수 있는 것으로 알려진 세가지 요소, 즉, 피험자 수, 가교문항의 길이 및 집단의 동등성에 따른 동등화 결과를 비교하고자 하였다. 이 연구에서는 문항반응이론 방법(이하 IRT)을 이용한 모의실험 자료를 이용하였으며 피험자 수의 조건, 가교문항의 길이, 집단의 동등성을 고려한 총 18가지의 조건을 고려하였으며 동등화 결과의 비교를 위해 IRT 동등화 결과를 준거로 삼아, 조건적 오차와 가중평균 오차 값을 산출하였다. 연구결과, 동등집단에서의 방법들 간 동등화 관계가 비동등집단에서의 방법들 간 동등화 관계보다 더 유사하게
나타났다. 하지만, 동등집단의 조건에서는 피험자 수와 가교문항의 길이에 상관없이 커넬의 선형 동등화 방법이 Tucker 선형 동등화 방법보다 적은 양의 가중평균 오차 값을 산출하였으며, 빈도추정동등화 방법이 커넬의 동백분위 동등화 방법보다 적은 양의 가중평균 오차 값을 산출하였다. 비동등집단 조건 중 충분한(30% 이상) 수의 가교문항이 사용된 경우, 피험자 수와 상관없이 Tucker 선형 동등화 방법이 적은 양의 가중평균 오차 값을 산출하였고, 적은(20% 이하) 수의 가교문항이 사용된 경우에는 커넬의 선형 동등화 방법이 적은 양의 가중평균 오차 값을 산출하였다. 이와는 달리, 동백분위 방법들의 경우에서는 비동등집단 조건이라 하더라도 가교문항의 길이에 상관없이 적정 수준(1,500명) 이하의 피험자수 조건에서는 전통적 동등화 방법인 빈도추정 방법이, 많은 수의 피험자를 가진 조건에서는 커넬의 동백분위 동등화 방법이 각각 적은 양의 가중평균 오차 값을 산출하였다.

다국어 초록 (Multilingual Abstract)

This study investigated the effect of the sample size, the anchor test length, and the group equivalency on the kernel and the traditional equating methods under the non-equivalent groups of anchor test (NEAT) design. The present study considered 18 conditions using simulated test data with three traditional equating methods and two post-stratification kernel equating methods were applied. The IRT equating results were used as the true criterion to compare the equating results of the kernel and the traditional methods. Like other studies, the equating relationships between the kernel linear and Tucker linear, and between the kernel equipercentile and the frequency estimation method became more similar under the equivalent groups conditions. Under the equivalent groups conditions, the kernel linear method generally yielded less amount of aggregate error than Tucker linear method regardless of sample sizes or anchor test lengths. The frequency estimation method also yielded less amount of aggregate error than kernel equipercentile equating method. However, under the NEAT conditions, Tucker linear would be preferred when there are enough anchor test items. In contrast, the kernel would be preferred with short anchor test items(less than 20% of total test length). Further, it is recommended to use the frequency estimation method with relatively small sample size (less than 1,500) while the kernel
equipercentile can be better with large sample size (more than 3,000) under the NEAT conditions. In short, the kernel and Tucker linear methods seemed to be more affected by the anchor test lengths while the frequency estimation and the kernel equipercentile methods seemed to be more affected by the sample sizes under the NEAT conditions.

목차 (Table of Contents)

• Ⅰ. Introduction
• Ⅱ. Kernel Equating Method
• Ⅲ. Method
• Ⅳ. Results
• Ⅴ. Conclusions

참고문헌 (Reference)

1 Han, K. T., "WinGen2 [Computerprogram]. Amherst" University of Massachusetts, Center for Educational Assessment 2007

2 Livingston, S. A., "What combination of sampling and equating methods works best?" 3 : 73-95, 1990

3 Holland, P. W., "The kernel method of equating score distributions" Educational Testing Service 1989

4 Ricker, K. L., "The impact of anchor test length on equating results in a nonequivalent groups design" Educational Testing Service 2007

5 von Davier, A. A., "The Kernel method of test equating" Springer-Verlag 2004

6 Holland, P. W., "Testing the untestable assumptions of the chain and post-stratification equating methods for the NEAT design" Educational Testing Service 2006

7 Kolen, M. J., "Test equating. in: Methods and practices" Springer 1995

8 Kolen, M. J., "Test equating, scaling, and linking. in: Methods and practices" Springer 2004

9 Livingston, S. A., "Small-sample equating with log-linear smoothing" 30 : 23-39, 1993

10 Liu, J., "Score linking issues related to test content changes. in: Linking and aligning scores and scales" Springer Science and Business Media, LLC 109-134, 2007

11 Angoff, W. H., "Scales, norms, and equivalent scores. in: Educational measurement" American Council on Education 1971

12 von Davier, A. A., "Population invariance and linear equating for the non-equivalent groups design" Educational Testing Service 2004

13 Hanson, B., "PIE [Computerprogram]" Iowa Testing Programs 2004

14 Dorans, N. J., "Linking and aligning scores and scales" Springer Science and Business Media, LLC 2007

15 ETS, "KE-software [Computer program]. Princeton" ETS 2007

16 Livingston, S. A., "Equating test scores (without IRT)" Educational Testing Service 2004

17 Kolen, M. J., "Does matching in equating work? A discussion" 3 : 97-104, 1990

18 Mao, X., "Comparisons of the kernel equating method with the traditional equating methods on Praxis™ data" Educational Testing Service 2006

19 Sinharay, S., "Choice of anchor test in equating" Educational Testing Service 2006

20 Kolen, M. J., "CIPE version 2.0 [Computer program]" Iowa Testing Programs 2004

21 Dorans, N. J., "Anchor test type and population invariance: An exploration across subpopulations and test administrations" 32 : 81-97, 2008

22 Liu, J., "An exploration of kernel equating using SAT® data: Equating to a similar population and to a distant population" Educational Testing Service 2007

23 Grant, M. C., "An evaluation of kernel equating: parallel equating with classical methods in the SAT subject tests™ program" Educational Testing Service 2009

24 Kim, S., "An alternative to equating with small samples in the non-equivalent groups anchor test design" Educational Testing Service 2006

25 Kolen, M. J., "An NCME instructional module on traditional equating methodology" 7 : 29-36, 1988

26 Liu, J., "A comparison of the kernel equating method with traditional equating methods using SAT® data" 45 : 309-323, 2008

27 Wang, T., "A comparison of the frequency estimation and chained equipercentile methods under the common-item nonequivalent groups design" 32 : 632-651, 2008