순차적 레이블링을 위한 ACE와 Self-Attention을 활용한 인공 신경망에 관한 연구

국문 초록 (Abstract)

최근 자연어 처리 분야의 순차적 레이블링을 위해 복수의 단어 임베딩 모델과 Recurrent Neural Network(RNN) 기반의 순차 모델을 이용한 인공 신경망이 보편적으로 사용되고 있다. 이로 인해 신경망 모델에 적합한 임베딩 벡터의 결합을 찾는 것과 순차 모델의 역전파 과정 중 발생하는 기울기 소실 문제를 해결하는 것은 중요한 연구 주제가 되었다. 본 논문은 복수의 임베딩 벡터들 중 일부를 사용하여 최적의 임베딩 결합을 찾는 Automated Concatenation of Embeddings(ACE)와 기울기 소실 문제를 해결하기 위해 셀프 어텐션을 활용하는 방법을 제안한다. 제안 방식의 성능을 검증하기 위해 순차적 레이블링 문제의 일종인 품사 태깅과 개체명 인식에서 다양한 데이터셋을 사용해 실험을 수행하였다. 그 결과 현재 최고의 성능을 내는 논문의 모델들 보다 향상된 성능을 보인다.

다국어 초록 (Multilingual Abstract)

Recently multiple word embedding models and sequence models based on RNN have been widely used for sequence labeling in the field of natural language processing. It is essential to find optimal embedding vectors for concatenation. Also, we should address the vanishing gradient problem during the backpropagation. This paper proposes a method using self-attention and ACE for solving the problems. We verified the performance of our proposed method on datasets for sequence labeling such as part of speech tagging and named entity recognition. Our proposed model shows better performance than the state of the art models.

목차 (Table of Contents)

• 요약
• Abstract
• 1. 서론
• 2. 이론적인 배경
• 3. 제안된 방법론
• 4. 실험 및 관련 연구와의 성능 비교
• 5. 실험 결과
• 6. 결론
• References

참고문헌 (Reference)

1 조세형, "이종의 통계정보를 이용한 품사 부착 기법" 한국지능시스템학회 18 (18): 501-505, 2008

2 진승희, "온톨로지 지식 기반 특성치를 활용한Bidirectional LSTM-CRF 모델의시퀀스 태깅 성능 향상에 관한 연구" 한국지능정보시스템학회 24 (24): 253-267, 2018

3 임미영, "도메인 정보가 강화된 워드 임베딩을 사용한 한국어 텍스트 생성" 한국지능시스템학회 29 (29): 142-147, 2019

4 김호승, "감성사전 정보가 결합된 특징통합벡터를 활용한 감성분석" 한국지능시스템학회 30 (30): 494-499, 2020

5 Zhilin Yang, "Xlnet: Generalized autoregressive pretraining for language understanding" 5753-5763, 2019

6 김정미, "Word2vec을 활용한 RNN기반의 문서 분류에 관한 연구" 한국지능시스템학회 27 (27): 560-565, 2017

7 Alexis Conneau, "Unsupervised Crosslingual Representation Learning at Scale" 2020

8 Yoshua Bengio, "The problem of learning long-term dependencies in recurrent networks" 1183-1188, 1993

9 David R So, "The evolved transformer" 2019

10 Abhyuday Jagannatha, "Structured prediction models for RNN based sequence labeling in clinical text" 856-865, 2016

11 Benjamin Heinzerling, "Sequence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evaluation" 273-291, 2019

12 Michihiro Yasunaga, "Robust Multilingual Part-of-Speech Tagging via Adversarial Training" 976-986, 2018

13 Farhad Nooralahzadeh, "Reinforcement-based denoising of distantly supervised NER with partial annotation" 225-233, 2019

14 Razvan Pascanu, "On the difficulty of training Recurrent Neural Networks" 1310-1318, 2013

15 Dzmitry Bahdanau, "Neural machine translation by jointly learning to align and translate" 2015

16 Guillaume Lample, "Neural architectures for named entity recognition" 260-270, 2016

17 Barret Zoph, "Neural architecture search with reinforcement learning" 2017

18 Emma Strubell, "Linguistically-informed self-attention for semantic role labeling" 5027-5038, 2018

19 Leyang Cui, "Hierarchically-Refined Label Attention Network for Sequence Labeling" 4115-4128, 2019

20 Jeffrey Pennington, "Glove: Global vectors for word representation" 1532-1543, 2014

21 Jeffrey L. Elman, "Finding structure in time" 14 (14): 179-211, 1990

22 Piotr Bojanowski, "Enriching word vectors with subword information" 5 : 135-146, 2017

23 Mingxing Tan, "Efficientnet:Rethinking model scaling for convolutional neural networks" 6105-6114, 2019

24 Tomas Mikolov, "Efficient Estimation of Word Representations in Vector Space"

25 Thang Luong, "Effective approaches to attention-based neural machine translation" 1412-1421, 2015

26 Matthew Peters, "Deep contextualized word representations" 2227-2237, 2018

27 Genta Indra Winata, "Deep contextualized word representations" 3541-3547, 2019

28 Ilya Loshchilov, "Decoupled Weight Decay Regularization" 2019

29 Chang Xu, "Cross-target stance classification with selfattention networks" 778-783, 2018

30 Alan Akbik, "Contextual String Embeddings for Sequence Labeling" 1638-1649, 2018

31 John Lafferty, "Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data" 282-289, 2001

32 Veysel Kocaman, "Biomedical Named Entity Recognition at Scale"

33 Shreyas Sharma, "BioFLAIR:Pretrained Pooled Contextualized Embeddings for Biomedical Sequence Labeling Tasks"

34 Jiao Li, "BioCreative V CDR task corpus: a resource for chemical disease relation extraction" 2016

35 Alex Graves, "Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition" 799-804, 2005

36 Jacob Devlin, "BERT: Pre-training of deep bidirectional transformers for language understanding" 4171-4186, 2019

37 Xinyu Wang, "Automated Concatenation of Embeddings for Structured Prediction"

38 Ashish Vaswani, "Attention is all you need" 6000-6010, 2017

39 Natalia Silveira, "A gold standard dependency corpus for English" 2014

40 John F. Kolen, "A Field Guide to Dynamical Recurrent Networks" Wiley-IEEE Press 2001