Angoff 방법과 북마크 방법을 활용한 한국어능력시험의 기준 설정에 관한 연구 = (A)study on setting the standard for test of proficiency in Korean which applies Angoff method and Bookmark method

다국어 초록 (Multilingual Abstract)

TOPIC(Test of Proficiency in Korean), which was introduced in 1997, is a criterion-referenced test that evaluates the Korean communication ability in the Korean language for those who learn Korean language as foreign language and has become increasingly important as a growing number of applicants from all over the region are submitting the application for the test every year.
The success or failure of criterion-referenced test is determined by the standard which provides the basis for the decision-making, and the right standard plays a very significant role in the criterion-referenced test. Nonetheless, whether an applicant passes or fails in the test is determined by the same standard that requires over 60 mean scores in overall area and 40 scores by section.

This standard is considered never reasonable from the evaluational viewpoint, given that 60% right answer rate in the overall area and 40% right answer rate in each section of the test do not validate that the applicant is equipped with the minimum communication skill in the Korean language within the targeted range.
In this regard, this study aimed to review the basis scores that provide the standard to determine whether the applicant passed or failed in the test, and explore the way to set the standard for presenting more reasonable and reliable basis in an attempt to facilitate the TOPIC(Test of Proficiency in Korean) to be established as a reliable criterion-referenced test tool to assess the fluency in Korean and given an officially social recognition that suits the purpose of the test.
In this study, the ''Listening'' and ''Reading'' comprehension section of the "9th TOPIC(Test of Proficiency in Korean) for third degree" in 2005 was selected as the standard test for setting the criteria, and 13 research associates were picked up who were teaching and researching those that were learning Korean as foreign langue in Korean education institutes. Those selected research associates were later trained on the standard setting. Following those procedures, the research was carried out based on the Angoff method and Bookmark method.


The findings of this study indicated the followings:

First, the standard deviation narrowed and reached a stable condition throughout the listening comprehension and the reading comprehension section as each test session proceeded when the Angoff method was employed, whereas the gap between the standard deviation and the standard test score by test session widened when the Bookmark method was applied.
In comparison with the final standard scores, the test scores differed somewhat depending on the method of standard setting, and the Angoff method led to higher conversion test scores than the Bookmark method did. It turned out that final conversion scores in the reading comprehension test were slightly higher when the Bookmark method was used, but the difference in test scores was not noticeable.

Second, By categorizing applicants who stood or failed in the test through the standard scores under each method, and applying the consistent statistics PA and Kappa coefficient between two methods by each round, it was indicated that the round''s consistency of Angroff was higher and more stable than when the Bookmark method was applied and the rating consistency of the third and the fourth was very high in the listening comprehension and reading comprehension section under the both methods.

Third, both listening comprehension and the reading comprehension showed lower consistency by method in the first rating when the consistency among standard setters was compared. But the consistency among standard setters increased as each session proceeded, which suggested that Angoff method led to a higher and more stable consistency in the early part of the round compared to the Bookmark method.

Fourth, it turned out that the evaluation condition had to expand more compared to the Bookmark method, which was found when the theory on the generalizationability was applied in an effort to explore the optimalization with respect to the method for setting the standard.
In addition, it was found that both Angoff method and Bookmark method were capable of increasing the generalization coefficient by expanding the number of group rather than the number of standard setter.

Fifth, the Angoff is intelligently challenging and time-consuming in comparing the applicability of standard-setting method, but is more reasonable than the Bookmark method, and more suitable for applying to the listening and reading comprehension section.

This study could not deal with the vocabulary․ grammar, and writing section of TOPIC(Test of Proficiency in Korean) because one dimensionality of test, which is one of the hypotheses necessary to apply the Bookmark method, was not fulfilled.
Therefore, future studies need to review the vocabulary․ grammar section of TOPIC(Test of Proficiency in Korean) and discuss in relation to the categorization of section subject to the evaluation. Furthermore, efforts must be made to set the standard for the TOPIC which is suitable for the new system to be introduced in 2006.

Finally, this method of setting the standard needs to be applied to the real education of Korean language, as well as TOPIC, in order to create more reasonable standard for evaluation. Moreover, researches must be made more vigorously, making comparisons with other external materials and data to validate the reasonability of the standard derived from the standard-setting method.