Sequence-to-text 기법을 활용한 언어 모형의 소개 및 분류 성능 비교 = Comparison and application of sequence-to-text based language models

국문 초록 (Abstract)

최근에 활발한 언어 모형의 개발과 함께 많은 분야의 연구자들이 이에 관심을 갖게 되었다. 언어 모형의 사용처는 매우 다양하여 많은 분야에 사용될 수 있다. 전통적인 통계학이나 머신 러닝의 분야에서는 이진 분류에 대한 연구가 활발하게 진행되어 왔고, 언어 모형들 역시 동일한 과제를 수행할 수 있다. 언어 모형의 특징으로 인해 입력 데이터는 테이블 형식의 숫자로 이루어진 데이터가 아니라 문자 형식의 변수를 입력해야 한다. 이 논문에서는 열 가지의 최신 언어 모형들의 목적과 특징들을 소개한다. 한편, 언어 모형의 데이터 입력을 위해 시퀀스 데이터를 텍스트로 변환하는 sequence-to-text 방법의 여러가지 예시를 구체적으로 제시하고, 예제 코드가 담긴 링크도 제공한다. 이를 통해 언어 모형 구현에 경험이 없는 독자들에게는 도움이 될 것으로 기대한다. 마지막으로, 제시한 여러가지 sequence-to-text 방법을 실제 데이터를 이용한 모의 실험을 통하여 이 논문에서 소개하는 최신 언어 모형들의 성능을 비교하고 머신 러닝 모형들의 결과도 함께 제공하여 언어 모형들의 우수한 분류 성능을 확인한다.

다국어 초록 (Multilingual Abstract)

Recently, the active development of language models has garnered significant interest from researchers across various fields. The applications of language models are exceedingly diverse, making them suitable for a multitude of domains. In traditional statistics and machine learning, extensive research has been conducted on binary classification, a task that language models can also perform effectively. Due to the inherent characteristics of language models, the input data required is not merely numerical in tabular form, but rather comprises textual variables. This paper presents an overview of the objectives and features of ten state-of-the-art language models. Additionally, various examples of sequence-to-text methods for transforming sequence data into textual formats are detailed to assist readers who may lack experience in implementing language models. Finally, through a series of simulations utilizing actual data, this study compares the performance of the aforementioned modern language models across several sequence-to-text methods, while also providing results from machine learning models to demonstrate the superior classification performance of language models.

참고문헌 (Reference)

1 Wang, T., "player2vec : A language modeling approach to understand player behavior in games"

2 Tang, G., "Why self-attention? A targeted evaluation of neural machine translation architectures"

3 Xu, Z., "Using large pre-trained language models to assist FDA in premarket medical device"

4 이승언 ; 김현세 ; 김남호 ; 엄정윤 ; 우지환, "Study on the real owner distinction in financial transactions using deep learning" 32 : 781-797, 2021

5 Joshi, M., "Spanbert : Improving pre-training by representing and predicting spans" 8 : 64-77, 2020

6 Bhowmik, S., "Sentiment analysis with hotel customer reviews using FNet" 13 : 1298-1306, 2024

7 Liu, Y., "Roberta : A robustly optimized bert pretraining approach"

8 Kitaev, N., "Reformer: The efficient transformer" 2020

9 Mikolov, T., "Recurrent neural network based language model. Interspeech 2010" 1045-1048, 2010

10 권순보 ; 유진은, "Online career counseling text classification using BERT and FastText" 33 : 991-1006, 2022

11 Song, K., "MPNet: Masked and permuted pre-training for language understanding" 33 : 16857-16867, 2020

12 Loh, N. K. N., "MPNet-GRUs : Sentiment analysis with masked and permuted pre-training for language understanding and gated recurrent units" 12 : 74069-74080, 2024

13 Beltagy, I., "Longformer : The long-document transformer"

14 Tay, Y., "Long range arena : A benchmark for efficient transformers"

15 Wang, Y., "LoBERTa : A composition named entity recognition method based on longformer and DeBERTa model" 266-270, 2022

16 Wu, Y., "Google’s neural machine translation system : Bridging the gap between human and machine translation"

17 Dai, Z., "Funnel-transformer: Filtering out sequential redundancy for efficient language processing" 33 : 4271-4282, 2020

18 Riaz, N., "From transformers to reformers" 1-6, 2021

19 Lee-Thorp, J., "FNet : Mixing tokens with Fourier transforms"

20 Cortiz, D., "Exploring transformer models for emotion recognition: A comparison of BERT, Distil-BERT, RoBERTa, XLNET and ELECTRA" 230-234, 2022

21 Bojanowski, P., "Enriching word vectors with subword information" 5 : 135-146, 2017

22 Wu, S., "Enriching pre-trained language model with entity information for relation classification" 2361-2364, 2019

23 Chen, Q., "End-to-end user behavior retrieval in clickthrough rate prediction model"

24 Clark, K., "ELECTRA: Pre-training text encoders as discriminators rather than generators" 2020

25 Sanh, V., "DistilBERT, a distilled version of BERT : Smaller, faster, cheaper, and lighter"

26 Samadi, M., "Deep contextualized text representation and learning for fake news detection" 58 : 102723-, 2021

27 He, P., "DeBERTaV3 : Improving DeBERTa using ELECTRA-style pretraining with gradient-disentangled embedding sharing"

28 Brown, P. F., "Class-based n-gram models of natural language" 18 : 467-479, 1992

29 Zaheer, M., "Big bird: Transformers for longer sequences" 33 : 17283-17297, 2020

30 Lewis, M., "Bart : Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension"

31 Devlin, J., "BERT : Pre-training of deep bidirectional transformers for language understanding"

32 Vaswani, A., "Attention is all you need" 5998-6008, 2017

33 Liu, Z., "Anomaly detection of command shell sessions based on DistilBERT : Unsupervised and supervised approaches"

34 Zhang, S., "An emotional classification method of Chinese short comment text based on ELECTRA" 34 : 254-273, 2022

35 Gasparetto, A., "A survey on text classification algorithms : From text to predictions" 13 : 83-, 2023

36 Bengio, Y., "A neural probabilistic language model" 3 : 1137-1155, 2003