딥러닝의 모형과 응용사례

안성만(Ahn, SungMahn) 한국지능정보시스템학회 2016 지능정보연구 Vol.22 No.2

Deep learning model is a kind of neural networks that allows multiple hidden layers. There are various deep learning architectures such as convolutional neural networks, deep belief networks and recurrent neural networks. Those have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition and bioinformatics where they have been shown to produce state-of-the-art results on various tasks. Among those architectures, convolutional neural networks and recurrent neural networks are classified as the supervised learning model. And in recent years, those supervised learning models have gained more popularity than unsupervised learning models such as deep belief networks, because supervised learning models have shown fashionable applications in such fields mentioned above. Deep learning models can be trained with backpropagation algorithm. Backpropagation is an abbreviation for “backward propagation of errors” and a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent. The method calculates the gradient of an error function with respect to all the weights in the network. The gradient is fed to the optimization method which in turn uses it to update the weights, in an attempt to minimize the error function. Convolutional neural networks use a special architecture which is particularly well-adapted to classify images. Using this architecture makes convolutional networks fast to train. This, in turn, helps us train deep, muti-layer networks, which are very good at classifying images. These days, deep convolutional networks are used in most neural networks for image recognition. Convolutional neural networks use three basic ideas: local receptive fields, shared weights, and pooling. By local receptive fields, we mean that each neuron in the first(or any) hidden layer will be connected to a small region of the input(or previous layer’s) neurons. Shared weights mean that we’re going to use the same weights and bias for each of the local receptive field. This means that all the neurons in the hidden layer detect exactly the same feature, just at different locations in the input image. In addition to the convolutional layers just described, convolutional neural networks also contain pooling layers. Pooling layers are usually used immediately after convolutional layers. What the pooling layers do is to simplify the information in the output from the convolutional layer. Recent convolutional network architectures have 10 to 20 hidden layers and billions of connections between units. Training deep learning networks has taken weeks several years ago, but thanks to progress in GPU and algorithm enhancement, training time has reduced to several hours. Neural networks with time-varying behavior are known as recurrent neural networks or RNNs. A recurrent neural network is a class of artificial neural network where connections between units form a directed cycle. This creates an internal state of the network which allows it to exhibit dynamic temporal behavior. Unlike feedforward neural networks, RNNs can use their internal memory to process arbitrary sequences of inputs. Early RNN models turned out to be very difficult to train, harder even than deep feedforward networks. The reason is the unstable gradient problem such as vanishing gradient and exploding gradient. The gradient can get smaller and smaller as it is propagated back through layers. This makes learning in early layers extremely slow. The problem actually gets worse in RNNs, since gradients aren’t just propagated backward through layers, they’re propagated backward through time. If the network runs for a long time, that can make the gradient extremely unstable and hard to learn from. It has been possible to incorporate an idea known as long short-term memory units (LSTMs) into RNNs. LSTMs make it much easier to get good results when traini

그래프 합성곱-신경망 구조 탐색 : 그래프 합성곱 신경망을 이용한 신경망 구조 탐색

최수연,박종열 국제문화기술진흥원 2023 The Journal of the Convergence on Culture Technolo Vol.9 No.1

This paper proposes the design of a neural network structure search model using graph convolutional neural networks. Deep learning has a problem of not being able to verify whether the designed model has a structure with optimized performance due to the nature of learning as a black box. The neural network structure search model is composed of a recurrent neural network that creates a model and a convolutional neural network that is the generated network. Conventional neural network structure search models use recurrent neural networks, but in this paper, we propose GC-NAS, which uses graph convolutional neural networks instead of recurrent neural networks to create convolutional neural network models. The proposed GC-NAS uses the Layer Extraction Block to explore depth, and the Hyper Parameter Prediction Block to explore spatial and temporal information (hyper parameters) based on depth information in parallel. Therefore, since the depth information is reflected, the search area is wider, and the purpose of the search area of the model is clear by conducting a parallel search with depth information, so it is judged to be superior in theoretical structure compared to GC-NAS. GC-NAS is expected to solve the problem of the high-dimensional time axis and the range of spatial search of recurrent neural networks in the existing neural network structure search model through the graph convolutional neural network block and graph generation algorithm. In addition, we hope that the GC-NAS proposed in this paper will serve as an opportunity for active research on the application of graph convolutional neural networks to neural network structure search. 본 논문은 그래프 합성곱 신경망을 이용한 신경망 구조 탐색 모델 설계를 제안한다. 딥 러닝은 블랙박스로 학습이 진행되는 특성으로 인해 설계한 모델이 최적화된 성능을 가지는 구조인지 검증하지 못하는 문제점이 존재한다. 신경망 구조 탐색 모델은 모델을 생성하는 순환 신경망과 생성된 네트워크인 합성곱 신경망으로 구성되어있다. 통상의 신경망 구조 탐색 모델은 순환신경망 계열을 사용하지만 우리는 본 논문에서 순환신경망 대신 그래프 합성곱 신경망을 사용하여 합성곱 신경망 모델을 생성하는 GC-NAS를 제안한다. 제안하는 GC-NAS는 Layer Extraction Block을 이용하여 Depth를 탐색하며 Hyper Parameter Prediction Block을 이용하여 Depth 정보를 기반으로 한 spatial, temporal 정보(hyper parameter)를 병렬적으로 탐색합니다. 따라서 Depth 정보를 반영하기 때문에 탐색 영역이 더 넓으며 Depth 정보와 병렬적 탐색을 진행함으로 모델의 탐색 영역의 목적성이 분명하기 때문에 GC-NAS대비 이론적 구조에 있어서 우위에 있다고 판단된다. GC-NAS는 그래프 합성곱 신경망 블록 및 그래프 생성 알고리즘을 통하여 기존 신경망 구조 탐색 모델에서 순환 신경망이 가지는 고차원 시간 축의 문제와 공간적 탐색의 범위 문제를 해결할 것으로 기대한다. 또한 우리는 본 논문이 제안하는 GC-NAS를 통하여 신경망 구조 탐색에 그래프 합성곱 신경망을 적용하는 연구가 활발히 이루어질 수 있는 계기가 될 수 있기를 기대한다.

대규모 신경망의 관점에서 본 우울증

김양태(Yang-Tae Kim) 대한생물치료정신의학회 2017 생물치료정신의학 Vol.23 No.1

Recent developments in the emerging science of large-scale neural networks offer a new understanding of a coherent paradigm for cognition. The perspective of large-scale neural networks provides a powerful framework for investigating psychopathology in psychiatric disorders. In a similar vein, altered organizations in large-scale neural networks are shown to play a prominent role in depression. In this respect, this review gives an overview of a diverse literature on depression from the perspectives of large-scale neural networks. First, both definition and function of large-scale neural networks will be provided. Second, from a large-scale neural networks perspective, symptoms of depression will be discussed. Next, the relationship between psychodynamics of depression and altered organizations in large-scale neural networks will be addressed. Lastly, it will be explained how antidepressants and psychotherapy influence on large-scale neural networks. Understanding depression in terms of large-scale neural networks will be expected to provide a better option of treatment for depression.

Application of Neural Network Model to Vehicle Emissions

김대현,이정 서울시립대학교 도시과학연구원 2010 도시과학국제저널 Vol.14 No.3

The issue of air quality is now a major concern around the world and the vehicle emissions model is very important. Most of the current vehicle emission models are multiple regression techniques. In this study, a neural network-based model has been proposed to achieve better estimation accuracy. The estimation performance of two models, the proposed neural network-based model and a general regression model, has been compared using mean absolute error (MAE). A comparative study between two models to estimate vehicle emissions, the proposed neural network-based model and a general regression model, has been conducted to assess the estimation performance of the proposed model in terms of mean absolute percentage error. Experimental results in this study revealed that the neural network model performed better as it was able to decrease the error for emission estimation comparing with the multiple regression models. More importantly, in this study a lookup table (LUT) method has been proposed to overcome the black-box problem, which is a disadvantage of the neural network models. It could be useful for any other researches to estimate emissions without developing and training the neural network model which can be a time-consuming task. The issue of air quality is now a major concern around the world and the vehicle emissions model is very important. Most of the current vehicle emission models are multiple regression techniques. In this study, a neural network-based model has been proposed to achieve better estimation accuracy. The estimation performance of two models, the proposed neural network-based model and a general regression model, has been compared using mean absolute error (MAE). A comparative study between two models to estimate vehicle emissions, the proposed neural network-based model and a general regression model, has been conducted to assess the estimation performance of the proposed model in terms of mean absolute percentage error. Experimental results in this study revealed that the neural network model performed better as it was able to decrease the error for emission estimation comparing with the multiple regression models. More importantly, in this study a lookup table (LUT) method has been proposed to overcome the black-box problem, which is a disadvantage of the neural network models. It could be useful for any other researches to estimate emissions without developing and training the neural network model which can be a time-consuming task.

Application of Data Fusion Technology Based on Weight Improved Particle Swarm Optimization Neural Network Algorithm in Wireless Sensor Networks

Xiajun Ding,Hongbo Bi,Xiaodan Jiang,Lu zhang 보안공학연구지원센터 2016 International Journal of Future Generation Communi Vol.9 No.3

With the development of sensor technology, network technology, embedded control technology and wireless communication technology, the application of wireless sensor networks (WSN) has become more and more widely. Wireless sensor networks have been named the most influential and important technology of the world in twenty-first Century. In wireless sensor networks, data fusion is an important research branch. In this paper, a data prediction model of wireless sensor network based on weight improved particle swarm optimization neural network algorithm is proposed. In view of the deficiency of the traditional BP neural network model, this paper combines with the characteristics of the data prediction model, and the BP neural network model is improved and integrated. After that, we train the neural network's sample set, and add the momentum item to correct the weight, so that the neural network can be predicted more quickly and accurately. The main idea of this paper is to predict the future data based on the historical data which are collected by sensor nodes, so as to achieve the purpose of reducing the amount of data transmission in the network and saving the energy of nodes. Finally, the experimental results show that the improved particle swarm optimization algorithm based on weight improved particle swarm optimization neural network algorithm has higher accuracy than the multiple regression method and the grey prediction method. In addition, the method can be used to effectively save energy in wireless sensor data transmission.

입자군집 최적화를 이용한 SVM 기반 다항식 뉴럴 네트워크 분류기 설계

노석범(Seok-Beom Roh),오성권(Sung-Kwun Oh) 대한전기학회 2018 전기학회논문지 Vol.67 No.8

In this study, the design methodology as well as network architecture of Support Vector Machine based Polynomial Neural Network, which is a kind of the dynamically generated neural networks, is introduced. The Support Vector Machine based polynomial neural networks is given as a novel network architecture redesigned with the aid of polynomial neural networks and Support Vector Machine. The generic polynomial neural networks, whose nodes are made of polynomials, are dynamically generated in each layer-wise. The individual nodes of the support vector machine based polynomial neural networks is constructed as a support vector machine, and the nodes as well as layers of the support vector machine based polynomial neural networks are dynamically generated as like the generation process of the generic polynomial neural networks. Support vector machine is well known as a sort of robust pattern classifiers. In addition, in order to enhance the structural flexibility as well as the classification performance of the proposed classifier, multi-objective particle swarm optimization is used. In other words, the optimization algorithm leads to sequentially successive generation of each layer of support vector based polynomial neural networks. The bench mark data sets are used to demonstrate the pattern classification performance of the proposed classifiers through the comparison of the generalization ability of the proposed classifier with some already studied classifiers.

Geometry Characteristics Prediction of Single Track Cladding Deposited by High Power Diode Laser Based on Genetic Algorithm and Neural Network

Huaming Liu,Xunpeng Qin,Song Huang,Lei Jin,Yongliang Wang,Kaiyun Lei 한국정밀공학회 2018 International Journal of Precision Engineering and Vol.19 No.7

This paper aims to establish a correlation between the process parameters and geometrical characteristics of the sectional profile of the single track cladding deposited by high power diode laser with rectangle beam spot. By applying the genetic algorithm and back propagation neural network, a nonlinear model for predicting the geometry features of the single track cladding is developed. A full factorial design method is used to conduct the experiments, and the experimental results are chosen randomly as training dataset and testing dataset for the neural network. Three main input variables such as laser power, scanning speed, and powder thickness were considered. The performance of the genetic algorithm and back propagation artificial neural network was compared to that of the standard back propagation neural network. To improve the accuracy of the neural network, one-hidden-layer and double-hidden-layer neural network with different architectures were performed. Further, one-output and multi-output neural network are also trained and tested. The results indicate that, by using genetic algorithm, the prediction accuracy of the neural network is significantly improved. Meanwhile, the double-hidden-neural network has higher prediction accuracy than the one-hidden-layer-neural network, while the one-output-neural network has higher prediction accuracy than the multi-output-neural network.

딥러닝 신경망을 이용한 신용카드 부도위험 예측의 효용성 분석

윤종문 한국금융학회 2019 금융연구 Vol.33 No.1

This study aims to discuss the usefulness of the deep learning neural network and the possibility of the deep learning neural network analysis in judging credit information by using credit card default data. Deep learning neural network analysis in the financial sector excluding the current stock price prediction model is under limited research. It is mainly used for upgrading models of the credit rating (Kvamme et al., 2016, 2018; Tran, 2016; Luo, 2017) and the delinquency rate (Sirignano et al., 2018). In the credit card market, it is focused on credit card issuance and fraud detection model (Ramanathan, 2014, Niimi, 2015). As mentioned earlier, there has not been much analysis of deep learning neural network using financial market data. This is because the study of deep learning neural networks is actively carried out mainly in the field of computer science such as image, speech recognition, natural language processing. Additionally, Researchers in the financial sector have difficulty learning deep learning algorithms and setting up a computer runtime environment. It is also difficult to apply the algorithm to financial data due to lower dimension than the image. Nowadays, financial companies have been interested in machine learning and are increasing their recruitment, but it is still in the stage of verifying the possibility of deep learning neural network. Therefore, This study examines the possibility of improving the accuracy of credit card default risk prediction by using a deep learning neural network algorithm. To do this, we use existing machine learning algorithms (Logistic, SVM, Random Forest, Lasso, etc.) as a comparison index for performance check of deep learning neural network analysis. Firstly, the deep learning neural network is constructed with two hidden layers and five neurons, and derives the prediction accuracy according to the activation function and the initial value setting method. There are Sigmoid, ReLU, tanh and Maxout as active functions, and random value, Xavier, RBM, He’s as initialization methods. Based on this, we compare the accuracy of existing machine learning algorithms. As a result, the deep learning neural network analysis showed performance improvement between 0.6% and 6.6%p compared to the existing machine learning algorithms (Logistic, SVM, Random Forest, Lasso, etc.). Among these results, the active function and the initial value setting method with the highest prediction accuracy are ReLU (rectified linear units) and Xavier initialization. However, there is no significant improvement in performance with increasing number of hidden layers and neurons up to 10 and 25, respectively. Also, the dropout and CNN (convolution neural network) models, which showed high performance in the field of image identification, showed no significant difference in prediction accuracy. Nevertheless, it could be interpreted that the increase of hidden layers can improve the accuracy of estimation because the highest accuracy (0.8161) and the AUC (0.7726) are observed for 10 hidden nodes and 15 neurons. However, we can’t say that accuracy increases linearly by the number of hidden layers and neurons. These limitation could be due to the quantitative and qualitative limitations of the credit card data used here. We did not use recurrent neural network (RNN) and long-short term memory (LSTM) models since the personal default data for credit card used in this study is cross-sectional data. These method are for Time-Series data. Therefore, it is expected that it will be able to obtain better results in identification problems (credit rating, delinquency rate, interest rate calculation) of present various financial markets if these deep learning neural network methodologies are applied through big data including time series data. This study can be turned into a question of how deep learning analysis can lower the default risk and delinquency rate by using financial data from a practical point of ... 본 연구는 국내․외 금융시장에서 아직 활성화되지 못한 딥러닝 신경망(deep learning neural network) 알고리즘을 이용해 신용카드 부도위험 예측의 정확도 향상 가능성에 대해서 점검한다. 이를 위해 기존 머신러닝 알고리즘(Logistic, SVM, Random Forest, Lasso 등)을 딥러닝 신경망 분석의 성능 점검을 위한 비교 지표로 활용한다. 우선, 딥러닝 신경망은 두 개의 은닉층(hidden layers)과 다섯 개의 뉴런(neuron)으로 구축하고, 활성함수(activation function)와 초기값(initial value) 설정방법에 따른 예측정확도를 도출한다. 그 결과 딥러닝 신경망 분석이 기존 머신러닝 알고리즘 보다 최소 0.6%p에서 최대 6.6%p 성능이 향상된 것으로 나타났다. 이 중 가장 높은 예측 정확도를 보인 활성함수와 초기값 설정방식은 ReLU(rectified linear units)와 Xavier(2010)이고 이를 기준으로 은닉층과 뉴런의 수를 각각 최대 10개와 25개까지 늘려 분석한 결과에서도 유사한 결과가 나타났다. 다만, 기존 연구에서와 같이 은닉층과 뉴런의 수의 증가에 따른 뚜렷한 성능의 향상은 나타나지 않았다. 또한, 이미지 식별 분야에서 높은 성능을 보였던 Dropout과 CNN(convolution neural network) 모델도 예측 정확도에서 큰 차이를 보이지 않았다. 이는 여기에서 사용된 신용카드 데이터가 다수 픽셀(pixel)로 이루어진 이미지 데이터와 비교해 양적․질적 한계가 있기 때문으로 판단된다. 한편, 본 연구에서 사용된 개인의 신용카드 부도 데이터는 횡단면 자료이기 때문에 시계열 데이터에서 높은 성능을 나타내는 RNN(recurrent neural network) 및 LSTM(Long- Short Term Memory) 등의 딥러닝 신경망 알고리즘을 사용하지는 않았다. 따라서 추후 시계열 자료가 포함된 빅데이터를 통해 이들 딥러닝 신경망 방법론을 적용한다면, 현재의 다양한 금융시장의 식별문제(신용등급, 연체율, 금리산정)에 있어 보다 향상된 결과를 도출할 수 있을 것으로 기대된다.

딥러닝을 이용한 다중이용시설의 이용자수 조사

정지헌(Jung, Ji Heon),박자연(Park, Ja Yeon),한수희(Han, Soohee) 대한토목학회 2022 대한토목학회 학술대회 Vol.2022 No.10

신경망이론을 이용한 폴리우레탄 코팅포 촉감의 예측

이정순,신혜원 한국의류학회 2002 한국의류학회지 Vol.26 No.1

Neural networks are used to predict the sense of touch of polyurethane coated fabrics. In this study, we used the multi layer perceptron (MLP) neural networks in Neural Connection. The learning algorithm for neural networks is back-propagation algorithm. We used 29 polyurethane coated fabrics to train the neural networks and 4 samples to test the neural networks. Input variables are 17 mechanical properties measured with KES-FB system, and output variable is the sense of touch of polyurethane coated fabrics. The influence of MLP function, the number of hidden layers, and the number of hidden nodes on the prediction accuracy is investigated. The results were as follows: MLP function, the number of hidden layer and the number of hidden nodes have some influence on the prediction accuracy. In this work, tangent function, the architecture of the double hidden layers and the 24-12-hidden nodes has the best prediction accuracy with the lowest RMS error. Using the neural networks to predict the sense of touch of polyurethane coated fabrics has better prediction accuracy than regression approach used in our previous study.