한국어 학습자의 진급 여부 의사결정을 위한 준거점수 설정 방법 비교 연구 = Establishing Passing Standards for Classroom Achievement Tests for Learners of Korean as Second Language: A Comparative Study of Three Methods.

다국어 초록 (Multilingual Abstract)

The purpose of the present study is to compare cut scores for classroom achievement tests at Korean language institutes using three well-established standard setting methods, and to investigate what factors may have an influence upon creating standards. In doing so, cut scores obtained by the extended Angoff's method, the Ebel's method and the contrasting groups method were compared with the exiting official cut scores based on experienced teachers' intuition at an university-affiliated language institute for learners of Korean as second language (KSL). The results indicate that the extended Angoff's and the Ebel's passing scores were lower than both the official and the contrasting groups' ones although the contrasting groups' passing scores produced similar results to the official ones. More intriguing is a significant change in cut scores and improvement in interrater reliability found when the judges were provided with normative information on learners' performance. The forementioned findings suggest that the intuition-driven cut score setting method, which has been prevailing in the Korean education field, might have to be reconsidered, and using well-established standard setting methods together could help faculty to make a more informed and valid decision on which should pass or fail in the setting of KSL classrooms. Implications for further research and limitations of the present study were also discussed.

참고문헌 (Reference)

1 법무부 출입국, 법무부 출입국․외국인정책본부 1-35, 2011

2 Skorupski, W. P., "What are panelists really thinking when they participate in standard setting studies?" 18 (18): 233-256, 2005

3 Wang, N, "Use of the Rasch IRT model in standard setting: An item-mapping method" 40 (40): 231-253, 2003

4 Norcini, J. J., "The effect of various factors on standard setting" 25 (25): 57-65, 1988

5 Impara, J. C., "Teachers' ability to estimate item difficulty: A test of the assumptions in the Angoff standard setting method" 35 (35): 69-81, 1998

6 Struyf, E., "Studies in Educational Evaluation" 27 (27): 215-238, 2001

7 AERA, "Standards for educational and psychological testing" Washington: American Educational Research Association 1999

8 Glass, G. V., "Standards and criteria" 15 (15): 237-261, 1978

9 Bunch, M. B., "Standard setting: A guide to establishing and evaluating performance standards on tests" Thousand Oaks, CA: Sage Publications 2007

10 Mehrens, W. A., "Standard setting and the public good: Benefits accrued and anticipated, In Setting performance standards: Concepts, methods, and perspectives" Mahwah, NJ: Erlbaum 477-486, 2001

11 Hambleton R. K, "Setting performance standards, In Educational measurement" ACE 433-470, 2006

12 Hambleton, R. K., "Setting performance standards on complex educational assessments" 24 (24): 355-366, 2000

13 Meara, K. C., "Setting and validating standards on professional licensure and certification exams: A survey of current practices" 12 : 17-23, 2001

14 Cizek, G. J., "Setting Performance Standards: Contemporary Methods" 23 (23): 31-50, 2004

15 Angoff,W.H, "Scales, norms, and equivalent scores, In Educational measurement" Washington, DC: American Countil on Education 1971

16 Nijlen, D., "Modeling judgments in the Angoff and contrasting-groups method of standard setting" 45 (45): 45-63, 2008

17 Sireci S. G, "Measurement practices in national licensing examination programs: A survey" 3 (3): 21-25, 1992

18 Mehrens, A., "Measurement and evaluation in education and psychology" Orlando, FL: Harcourt Brace Jovanovich 1991

19 Linn, R. L., "Item-by-item standard setting: Misinterpretations of judge's intentions due to less than perfect item inter-correlations" 1997

20 Ferdous, A. A, "Item response theory-based approaches for computing minimum passing scores from an Angoff-based standard-setting study" 1 (1): 1-19, 2008

21 Green, D. R., "Interpreting the results of three different standard-setting procedures" 22 (22): 22-32, 2003

22 Stiggins, R. J., "Inside high school grading practices: Building a research agenda" 8 (8): 5-14, 1989

23 Busch, J. C., "Influence of type of judge, normative information, and discussion on standards recommended for the national teacher examinations" 27 (27): 145-163, 1990

24 Shepard, L. A., "Implications for standard setting of the National Academy of Education Evaluationof the National Assessment of Educational Progress Achievement Levels" 1995

25 Plake, B. S., "Factors influencing intrajudge consistency during standard-setting" 10 (10): 15-25, 1991

26 Mills, C. N., "Estimating and adjusting cutoff scores: Features of selected methods" 1 (1): 261-275, 1988

27 Downing, S. M., "Establishing passing standards for classroom achievement tests in medical education: A comparative study of four methods" 78 (78): S85-S87, 2003

28 Ebel, R. L., "Essentials of educational measurement" Englewood Cliffs, NJ: Prentice-Hall 1972

29 Field, A, "Discovering statistics using SPSS" London: SAGE Publications 2010

30 Berk,R.A, "Determination of optimal cutting scores in criterion-referenced measurement" 45 : 4-9, 1976

31 Brown, H. D., "Criterion-referenced language testing" Cambridge: Cambridge University Press 2002

32 Brandon, P. R, "Conclusions about frequently studied modified angoff standard-setting topics" 17 (17): 59-88, 2004

33 Reilly, R. R., "Comparison of direct and indirect methods for setting minimum passing scores" 8 (8): 421-429, 1984

34 Jaeger, R. M., "Certification of student competence" 3 : 485-514, 1989

35 Poggio, J. P., "An empirical investigation of the Angoff, Ebel, and Nedelsky standard setting methods" 1981

36 Cizek G.J, "An NCME instructional module on: Setting passing scores" 15 (15): 20-31, 1996

37 Nedelsky,L, "Absolute grading standards for objective tests" 14 (14): 3-19, 1954

38 Plake, B. S., "Ability of panelists to estimate item performance for a target group of candidates: An issue in judgmental standard setting" 7 (7): 87-97, 2001

39 Hurtz, G. M., "A meta-analysis of the effects of modifications to the Angoff method on cutoff scores and judgment consensus" 63 (63): 584-601, 2003

40 Buckendahl, C. W., "A comparison of Angoff and bookmark standard setting methods" 39 (39): 253-263, 2002

41 Impara, J C., "A Comparison of cut scores using, ultiple standard setting methods" 2000

42 Livingston, S. A., "A Comparative study of standard-setting methods" 2 (2): 121-141, 1989