Psychometric Properties of Composite Scores and Effects of Differential Weights Using Different Combining Methods

국문 초록 (Abstract)

이 연구에서는 두 개의 검사를 치른 수험생의 원점수를 표준점수 혹은 정답비율점수로 변환한 후 두 검사에 가중치를 주어 합산점수를 만든 후, 이것을 다시 척도 점수로 변환하는 상황에 초점을 두고 있다. 이 연구에서는 이와 같은 상황에서 합산된 원점수(composite raw score)와 척도점수(scale score)의 조건적 측정오차(conditional standard error of measurement: CSEM)와 신뢰도를 추정하는 절차, 그리고 척도점수의 분류일관성(classification consistency) 계수를 산출하는 수리적 절차를 제시하였다.
또한 두 검사에 가중치를 달리 주었을 때 이들 CSEM, 신뢰도 그리고 분류일관성 계수가 변화하는 정도를 실제 자료를 통해 보여 주었다. 두 검사 중 보다 신뢰로운 부분에 가중치를 줄수록 어느 정도까지는 CSEM이 작아지고 신뢰도는 높았다. 또한 두 검사에 다른 가중치를 줌에 따라 오차량이 변화하였으며, 때로는 변화된 오차의 크기가 상당히 컸다. 합산점수를 만들 때 표준점수를 사용하든 정답비율점수를 사용하든 CSEM과 신뢰도는 유사하였다.

다국어 초록 (Multilingual Abstract)

The situations described in this paper are ones in which examinees' number correct scores on two components are transformed to z-scores or proportion correct scores and then a differentially weighted sum is used to form examinees' composite raw scores. The composite raw scores are then transformed to scale scores. This paper described a procedure for estimating conditional standard errors of measurement (CSEM) and reliability for composite raw scores and scale scores, and for computing classification consistency for scale scores. This paper also illustrated how different methods for combining test scores and the choice of weights affect CSEMs and reliability for composite raw and scale scores, and classification consistency of scale scores. Putting more weight on the more reliable components produced smaller average CSEMs and higher reliabilities for the composite scores, up to the maximal weights of the two components. However, the amount of error in raw and scale scores was differentially, and sometimes substantially, increased or decreased with the different weights; the change in error for the different weights was larger for some scores than others. With the different weights and combining methods, classification consistencies also differed, sometimes substantially.

목차 (Table of Contents)

• ABSTRACT
• Ⅰ. Introduction
• Ⅱ. Estimation Methodology
• Ⅲ. Illustrative Example
• Ⅳ. Results
• Ⅴ. Conclusion and Discussion
• References
• FIGURE 1-9
• 초 록

참고문헌 (Reference)

1 "Weights that maximize reliability under a congeneric model for performance assessment" 22 : 179-187, 1998

2 "Weighting for battery reliability and prediction" 30 : 357-366, 1940

3 "Toward a Marxist theory of test construction Applied Measurement in Education" 103-118, 1993

4 "Standards for educational and psychological testing" American Educational Research Association 1999

5 "Procedures for computing classification consistency and accuracy indices with multiple categories(RR 2000-10)" ACT, Inc 2000

6 "Informed test component weighting" 20 (20): 16-19, 2001

7 "Estimators of conditional scale-score standard errors of measurement: A simulation study" 37 : 1-20, 2000

8 "Estimating the consistency and accuracy of classifications based on test scores" 32 : 179-197, 1995

9 "Distributions in statistics" 1972

10 "Conditional standard errors of measurement for scale scores using binomial and compound binomial assumptions" University of Iowa (41) : 1997

11 "Conditional standard errors of measurement for scale scores using IRT" 33 : 129-140, 1996

12 "Conditional standard errors of measurement for scale scores Journal of Educational Measurement" 285-307, 1992

13 "Conditional standard errors of measurement for composite scores using IRT" 1998

14 "Conditional scale-score standard errors of measurement under binomial and compound binomial assumptions" 59 : 5-24, 1999

15 "Complex composites: Issues that arise in combining different modes of assessment" 19 : 51-71, 1995

16 "An investigation of classification consistency indexes estimated under alternative strong true score models" 27 : 345-359, 1990

17 "A review of methods and empirical studies Review of Educational Research" 663-705, 1970

18 "A rational for defining achievement levels using IRT-estimated domain scores" 23 : 347-362, 1999

19 "A note on the calculation of weights for maximum battery reliability" 15 : 57-61, 1950

20 "A multinomial error model for tests with polytomous items" 2001