다중 레이블 분류의 정확도 향상을 위한 스킵 연결 오토인코더 기반 레이블 임베딩 방법론

다국어 초록 (Multilingual Abstract)

Recently, with the development of deep learning technology, research on unstructured data analysis is being actively conducted, and it is showing remarkable results in various fields such as classification, summary, and generation. Among various text analysis fields, text classification is the most widely used technology in academia and industry. Text classification includes binary class classification with one label among two classes, multi-class classification with one label among several classes, and multi-label classification with multiple labels among several classes. In particular, multi-label classification requires a different training method from binary class classification and multi-class classification because of the characteristic of having multiple labels. In addition, since the number of labels to be predicted increases as the number of labels and classes increases, there is a limitation in that performance improvement is difficult due to an increase in prediction difficulty.
To overcome these limitations, (i) compressing the initially given high-dimensional label space into a low-dimensional latent label space, (ii) after performing training to predict the compressed label, (iii) restoring the predicted label to the high-dimensional original label space, research on label embedding is being actively conducted. Typical label embedding techniques include Principal Label Space Transformation (PLST), Multi-Label Classification via Boolean Matrix Decomposition (MLC-BMaD), and Bayesian Multi-Label Compressed Sensing (BML-CS). However, since these techniques consider only the linear relationship between labels or compress the labels by random transformation, it is difficult to understand the non-linear relationship between labels, so there is a limitation in that it is not possible to create a latent label space sufficiently containing the information of the original label. Recently, there have been increasing attempts to improve performance by applying deep learning technology to label embedding. Label embedding using an autoencoder, a deep learning model that is effective for data compression and restoration, is representative. However, the traditional autoencoder-based label embedding has a limitation in that a large amount of information loss occurs when compressing a high-dimensional label space having a myriad of classes into a low-dimensional latent label space.
This can be found in the gradient loss problem that occurs in the backpropagation process of learning. To solve this problem, skip connection was devised, and by adding the input of the layer to the output to prevent gradient loss during backpropagation, efficient learning is possible even when the layer is deep. Skip connection is mainly used for image feature extraction in convolutional neural networks, but studies using skip connection in autoencoder or label embedding process are still lacking.
Therefore, in this study, we propose an autoencoder-based label embedding methodology in which skip connections are added to each of the encoder and decoder to form a low-dimensional latent label space that reflects the information of the high-dimensional label space well. In addition, the proposed methodology was applied to actual paper keywords to derive the high-dimensional keyword label space and the low-dimensional latent label space. Using this, we conducted an experiment to predict the compressed keyword vector existing in the latent label space from the paper abstract and to evaluate the multi-label classification by restoring the predicted keyword vector back to the original label space.
As a result, the accuracy, precision, recall, and F1 score used as performance indicators showed far superior performance in multi-label classification based on the proposed methodology compared to traditional multi-label classification methods. This can be seen that the low-dimensional latent label space derived through the proposed methodology we

참고문헌 (Reference)

1 조인수, "스킵연결이 적용된 오토인코더 모델의 클러스터링 성능 분석" 한국정보처리학회 9 (9): 403-410, 2020

2 Pascal, V, "Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion" 11 : 3371-3408, 2010

3 Bingyu, W., "Ranking-Based Autoencoder for Extreme Multi-label Classification" 1904

4 Denis, L, "On the use of binary stochastic autoencoders for multilabel classification under the zero-one loss" 144 : 71-80, 2018

5 Pierre, B., "Neural Networks and Principal Component Analysis: Learning from Examples Without Local Minima" 2 : 53-58, 1989

6 Farbound, T., "Multilabel Classification with Principal Label Space Transformation" 24 (24): 2508-2542, 2012

7 Ashish, K., "Multilabel Classification using Bayesian Compressed Sensing" 25 : 2012

8 Jörg, W, "Multi-label classification using boolean matrix decomposition" 179-186, 2012

9 Chih-Kuan, "Learning Deep Latent Space for Multi-Label Classification" 31 (31): 2017

10 Zichao Y, "Hierarchical Attention Networks for Document Classification" 1480-1489, 2016

11 Yoshua, B., "Greedy Layer-Wise Training of Deep Networks" 19 : 2007

12 Pascal, V, "Extracting and composing robust features with denoising autoencoders" 1096-1103, 2008

13 Tomas M, "Efficient Estimation of Word Representations in Vector Space" 2013

14 Kaiming, H., "Deep Residual Learning for Image Recognition" 770-778, 2016

15 Ashish, V., "Attention Is All You Need" 2017

16 Ganda, D., "A Survey on Multi Label Classification" 5 (5): 19-23, 2018

17 Jörg, W, "A Nonlinear Label Compression and Transformation Method for Multi-label Classification Using Autoencoders" 328-340, 2016