Ternary Decomposition and Dictionary Extension for Khmer Word Segmentation

다국어 초록 (Multilingual Abstract)

In this paper, we proposed a dictionary extension and a ternary decomposition technique to improve the effectiveness of Khmer word segmentation. Most word segmentation approaches depend on a dictionary. However, the dictionary being used is not fully reliable and cannot cover all the words of the Khmer language. This causes an issue of unknown words or out-of-vocabulary words. Our approach is to extend the original dictionary to be more reliable with new words. In addition, we use ternary decomposition for the segmentation process. In this research, we also introduced the invisible space of the Khmer Unicode (char\u200B) in order to segment our training corpus. With our segmentation algorithm, based on ternary decomposition and invisible space, we can extract new words from our training text and then input the new words into the dictionary. We used an extended wordlist and a segmentation algorithm regardless of the invisible space to test an unannotated text. Our results remarkably outperformed other approaches. We have achieved 88.8%, 91.8% and 90.6% rates of precision, recall and F-measurement.

목차 (Table of Contents)

• Abstract
• 1. Introduction
• 2. Khmer Language Overview
• 3. Research Reviews
• 4. Proposed Approach
• 5. Experiment
• 6. Conclusion
• References

참고문헌 (Reference)

1 Chea, S., "Word Bigram Vs Orthographic Syllable Bigram in Khmer Word Segmentation"

2 Seng, S., "Which Units for acoustic and language modelling for Khmer automatic speech recognition?"

3 Van, C., "Query Expansion for Khmer Information Retrieval" 80-87, 2010

4 Seng, S., "Multiple Text Segmentation for Statistical Language Modelling" France 2MICA Center 2009

5 Mohri, M. F., "Lecture Notes in Computer Science" Springer 144-158, 1998

6 Channa, V., "Khmer Word Segmentation and Out -of-Vocabulary Words Detection Using Collection Measurement of Repeated Characters Subsequences"

7 Khin, S., "Khmer Grammar" Royal Academy of Cambodia 2007

8 "Khmer Dictionary" Royal Academy of Cambodia 2005

9 Nevill-Manning, C. G., "Identifying Hierarchical Structure in Sequences A linear-time algorithm" 7 (7): 67-82, 1997

10 Nou, C., "Hybrid Approach for Khmer Unknown Word POS Guessing"

11 Puthick, H., "Development of a Khmer Spell Checker Based on a Hidden Markov Model" Australian National University 2005

12 Chea, S., "Detection and Correction of Homophonous Error Word for Khmer Language"

13 Thanopoulos, A., "Comparative Evaluation of Collocation Extraction Metrics"

14 Huffman, F. E., "Cambodian System of Writing and beginning reader with Drills and Glossary" Yale University Press 1970

15 Seng, S., "Boosting N-gram Coverage for Unsegmented Languages Using Multiple Text Segmentation Approach" 1-7, 2010

16 Church, K. W., "A Status Report on ACL/DCL" 84-91, 1991

17 Shannon, E., "A Mathematical Theory of Communication" 27 : 379-423, 1948