문서 분류를 위한 신경망 모델에 적합한 텍스트 전처리와 워드 임베딩의 조합

다국어 초록 (Multilingual Abstract)

Neural networks with word embedding have recently used for document classification. Researchers concentrate on designing new architecture or optimizing model parameters to increase performance. However, most recent studies have overlooked text preprocessing and word embedding, in that the description of text preprocessing used is insufficient, and a certain pretrained word embedding model is mostly used without any plausible reasons. Our paper shows that finding a suitable combination of text preprocessing and word embedding can be one of the important factors required to enhance the performance. We conducted experiments on AG’s News dataset to compare those possible combinations, and zero/random padding, and presence or absence of fine-tuning. We used pretrained word embedding models such as skip-gram, GloVe, and fastText. For diversity, we also use an average of multiple pretrained embeddings (Average), randomly initialized embedding (Random), task data-trained skip-gram (AGNews-Skip). In addition, we used three advanced neural networks for the sake of generality. Experimental results based on OOV (Out Of Vocabulary) word statistics suggest the necessity of those comparisons and a suitable combination of text preprocessing and word embedding.

참고문헌 (Reference)

1 "word2vec: tool fot computing continuous distributed representations of words"

2 "fastText: wiki word vectors"

3 Conneau, Alexis, "Very deep convolutional networks for natural language processing"

4 Lai, Siwei, "Recurrent Convolutional Neural Networks for Text Classification" 333 : 2015

5 Camacho-Collados, Jose, "On the Role of Text Preprocessing in Neural Network Architectures: An Evaluation Study on Text Categorization and Sentiment Analysis"

6 Chen, Huimin, "Neural sentiment classification with user and product attention" 2016

7 Bahdanau, Dzmitry, "Neural machine translation by jointly learning to align and translate"

8 Loper, Edward, "NLTK: The natural language toolkit" Association for Computational Linguistics 2002

9 Lei, Tao, "Molding cnns for text: non-linear, non-consecutive convolutions"

10 Hochreiter, Sepp, "Long short-term memory" 9 (9): 1735-1780, 1997

11 Wen, Ying, "Learning text representation using recurrent convolutional neural network with highway layers"

12 Yang, Zichao, "Hierarchical attention networks for document classification" 2016

13 Kowsari, Kamran, "Hdltex: Hierarchical deep learning for text classification"

14 Pennington, Jeffrey, "Glove: Global vectors for word representation" 2014

15 "GloVe: Global Vectors for Word Representation"

16 Raffel, Colin, "Feed-forward networks with attention can solve some long-term memory problems"

17 Bojanowski, Piotr, "Enriching word vectors with subword information"

18 Xiao, Yijun, "Efficient character-level document classification by combining convolution and recurrent layers"

19 Tang, Duyu, "Document modeling with gated recurrent neural network for sentiment classification" 2015

20 Mikolov, Tomas, "Distributed representations of words and phrases and their compositionality, Advances in neural information processing systems" 2013

21 Johnson, Rie, "Deep pyramid convolutional neural networks for text categorization" 1 : 2017

22 Kim, Yoon, "Convolutional neural networks for sentence classification"

23 Zhang, Xiang, "Character-level convolutional networks for text classification;Advances in neural information processing systems" 2015

24 Zhang, Ye, "A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification"

25 Zhou, Chunting, "A C-LSTM neural network for text classification"