분류 정확도 향상을 위한 선택적 마스킹 기반 추가 사전 학습 기법

국문 초록 (Abstract)

최근 여러 자연어 처리 분야에서 사전 학습 언어 모델인 BERT를 활용하여 분석 과제에 최적화된 텍스트 표현을 추출하려는 연구가 활발하게 이루어지고 있다. 특히 BERT의 학습 방식 중 하나인 MLM(Masked Language Model)을 활용하여 도메인 정보 또는 분석 과제 데이터를 추가 사전 학습(Further Pre-training)하는 시도가 이어지고 있다. 하지만 기존의 MLM 기법이 채택한 무작위 마스킹을 사용하여 감성 분류 과제에서 추가 사전 학습을 수행하는 경우, 분류 학습에 중요한 단서가 되는 단어가 마스킹될 수 있다는 가능성으로 인해 문장 전체에 대한 감성 정보 학습이 충분히 이루어지지 않는다는 한계가 있다. 이에 본 연구에서는 무작위 마스킹이 아닌 단서 단어를 제외하고 마스킹하는 선택적 마스킹을 통해 감성 분류 과제에 특화된 추가 사전 학습을 수행할 수 있는 방법을 제안한다. 더불어 주변 단어를 선택하기 위해 어텐션 메커니즘(Attention Mechanism)을 활용하여 단어의 감성 기여도를 측정하는 방안도 함께 제안한다. 제안 방법론을 실제 감성 댓글에 적용하여 문장 벡터를 추론하고 감성 분류 실험을 수행한 결과, 제안 방법론이 기존의 여러 비교 모델에 비해 분류 정확도 측면에서 우수한 성능을 나타냄을 확인하였다.

다국어 초록 (Multilingual Abstract)

Recently, studies to extract text expressions optimized for analysis tasks by utilizing bidirectional encoder representations from transformers (BERT), which is a pre-training language model, are being actively conducted in various natural language processing fields. In particular, attempts are being made to further pre-train domain information or target data using masked language model (MLM), which is one of the BERT training methods. However, if further pre-training is performed with the existing random masking when performing sentiment classification, there is a limitation that sentimental nuance for the entire sentence may not be sufficiently learned if the words that are important clues to the sentiment classification are masked. Therefore, in this study, we propose an further pre-training method specialized for sentiment classification tasks which sufficiently reflect sentiment information in sentences by selective masking that excludes clue words from masking candidates. In addition, this study proposes a method to distinguish between clue words and surrounding words as the role of words by utilizing the attention mechanism. On inferring sentence vectors by applying the proposed methodology to actual sentiment comments and performing sentiment classification experiments, it was confirmed that the proposed methodology showed superior performance in terms of classification accuracy compared to several existing comparison models.

목차 (Table of Contents)

• 요약
• Abstract
• 1. 서론
• 2. 관련 연구
• 3. 제안 방법
• 4. 실험
• 5. 결론
• References

참고문헌 (Reference)

1 "https://github.com/e9t/nsmc"

2 Y. Gu, "Train No Evil: Selective Masking for Task-guided Pre-training" 6966-6974, 2020

3 M. Joshi, "SpanBERT: Improving Pre-training by Representing and Predicting Spans" 8 : 64-77, 2020

4 I. Sutskever, "Sequence to Sequence Learning with Neural Networks" 2 : 3104-3112, 2014

5 I. Beltagy, "SciBERT: A Pretrained Language Model for Scientific Text" 3615-3620, 2019

6 T. Mikolov, "Recurrent Neural Network Based Language Model" 2010

7 C. Sung, "Pre-training BERT on Domain Resources for Short Answer Grading" 6071-6075, 2019

8 V. D. Viellieber, "Pre-trained language models as knowledge bases for Automotive Complaint Analysis"

9 D. Bahdanau, "Neural Machine Translation by Jointly Learning to Align and Translate"

10 M. Ghazvininejad, "Mask-predict: Parallel Decoding of Conditional Masked Language Models" 6112-6121, 2019

11 X. Wu, "Mask and Infill: Applying Masked Language Model to Sentiment Transfer" 5271-5277, 2019

12 S. Hochreiter, "Long Short-Term Memory" 9 (9): 1735-1780, 1997

13 J. B. Lee, "KcBERT: Korean Comments BERT" 437-440, 2020

14 A. Radford, "Improving Language Understanding by Generative Pre-training"

15 P. Bojanowski, "Enriching Word Vectors with Subword Information" 5 : 135-146, 2017

16 J. Chung, "Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling"

17 T. Mikolov, "Efficient Estimation of Word Representations in Vector Space" 2013

18 M. Peter, "Deep Contextualized Word Representations" 1 : 2227-2237, 2018

19 K. Huang, "Clinical-BERT: Modeling Clinical Notes and Predicting Hos pital Readmission"

20 J. Lee, "BioBERT: A Pre-trained Biomedical Language Representation Model for Biomedical Text Mining" 36 (36): 1234-1240, 2019

21 J. Devlin, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding"

22 Y. Wang, "Attention-based LSTM for Aspect-level Sentiment Classification" 606-615, 2016

23 A. Vaswani, "Attention Is All You Need" 6000-6010, 2017