사전 학습된 한국어 BERT의 전이학습을 통한 한국어 기계독해 성능개선에 관한 연구

다국어 초록 (Multilingual Abstract)

Language Models such as BERT has been an important factor of deep learning-based natural language processing. Pre-training the transformer-based language models would be computationally expensive since they are consist of deep and broad architecture and layers using an attention mechanism and also require huge amount of data to train. Hence, it became mandatory to do fine-tuning large pre-trained language models which are trained by Google or some companies can afford the resources and cost.
There are various techniques for fine tuning the language models and this paper examines three techniques, which are data augmentation, tuning the hyper paramters and partly re-constructing the neural networks. For data augmentation, we use no-answer augmentation and back-translation method. Also, some useful combinations of hyper parameters are observed by conducting a number of experiments. Finally, we have GRU, LSTM networks to boost our model performance with adding those networks to BERT pre-trained model.
We do fine-tuning the pre-trained korean-based language model through the methods mentioned above and push the F1 score from baseline up to 89.66. Moreover, some failure attempts give us important lessons and tell us the further direction in a good way.

참고문헌 (Reference)

1 Clark, K., "What Does BERT Look At? An Analysis of BERT’s Attention" Stanford University

2 Wang, R., "To Tune or not tune? How about the best of both worlds?" Percent Group, AI Lab

3 Ying, A., "Really Paying Attention : A BERT+ BiDAF Ensemble Model for Question-Answering" Standford University

4 Mohammadi, M., "Natural Language Processing With Deep Learning" Stanford University

5 임승영, "KorQuAD : 기계독해를 위한 한국어 질의응답 데이터셋" 539-541, 2018

6 Marivate, V., "Improving short text classification through global augmentation methods" 385-399, 2019

7 Sun, C., "How To Fine-Tune BERT For Text Classification?" Fudan University

8 Ethayarajh, K., "How Contextual Are Contextualised Word Representations? Comparing The Geometry of Bert, Elmo, And Gpt2" Stanford University

9 Dodge, J., "Fine-Tuning Pretrained Language Models : Weight Initializations, Data Orders, and Early Stopping" Cornell University

10 Qin, Z., "Diverse Ensembling with Bert and its variations for Question Answering on SQuAD 2.0"

11 Yang, W., "Data Augmentation for BERT Fine-Tuning in Open-Domain Question Answering"

12 Kobayashi, S., "Contextual Augmentation : Data Augmentation By Words With Paradigmatic Relations" Preferred Networks, Inc.

13 Lalande, K.M., "CS224n Final Project : SQuAD 2.0 with BERT"

14 Semnani, J.S., "BERTA : Fine-tuning BERT with Adapters and Data Augmentation" Standford University

15 Devlin, J., "BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding, Google AI Language"