확률 기반 미등록 단어 분리 및 태깅

다국어 초록 (Multilingual Abstract)

Processing of unknown words such as proper nouns and newly coined words is important for a morphological analyzer to process documents in various domains. In this study, a segmentation and tagging method for unknown Korean words is proposed for the 3-step probabilistic morphological analysis. For guessing unknown word, it uses rich suffixes that are attached to open class words, such as general nouns and proper nouns. We propose a method to learn the suffix patterns from a morpheme tagged corpus, and calculate their probabilities for unknown open word segmentation and tagging in the probabilistic morphological analysis model. Results of the experiment showed that the performance of unknown word processing is greatly improved in the documents containing many unregistered words.

참고문헌 (Reference)

1 이재성, "한국어 형태소 분석을 위한 3단계 확률 모델" 한국정보과학회 38 (38): 257-268, 2011

2 김선호, "한국어 문서 처리를 위한 동적 생성 로컬 사전 기반 미등록어 분석" 한국정보과학회 29 (29): 407-416, 2002

3 "http://wikitravel.org/ko/"

4 G. G. Lee, "Syllable-patternbased unknown-morpheme segmentation and estimation for hybrid part-of-speech tagging of Korean" 28 (28): 53-70, 2002

5 J. Kupiec, "Robust part-of-speech tagging using a hidden Markov model" 6 (6): 225-242, 1992

6 B.-R. Park, "Recognizing unknown words by analyzing their examples" 25 (25): 397-407, 1998

7 D. Lee, "Probabilistic models for Korean morphological analysis and part-of-speech tagging" Korea Univ. 2005

8 D. Lee, "Language model smoothing for Korean morpheme recovery" 39 (39): 309-311, 2012

9 S. S. Kang, "Korean morphological analysis using syllable information and multiple-word units" Seoul National University 1993

10 Y. Zhang, "Joint word segmentation and POS tagging using a single perceptron" 888-896, 2008

11 J. Hatori, "Incremental joint approach to word segmentation, pos tagging, and dependency parsing in chinese" 1 : 1045-1053, 2012

12 W. Gale, "Good-Turing smoothing without tears" 2 : 217-237, 1995

13 R. Weischedel, "Coping with ambiguity and unknown words through probabilistic models" 19 (19): 361-382, 1993

14 T. Nakagawa, "Chinese and Japanese word segmentation using word-level and character-level information" 466-472, 2004

15 B. Kim, "Automatic space information extraction from Korean text" 18 (18): 2953-2962, 2015

16 R. Sproat, "A stochastic finite-state word-segmentation algorithm for Chinese" 22 (22): 377-404, 1996

17 Y. Zhang, "A fast decoder for joint word segmentation and POS-tagging using a single discriminative model" Association for Computational Linguistics 2010

18 The National Institute of the Korean Language, "21st Century Sejong Project Final Result - 2011.12. Revised Edition"