GeoGebra 로 설명하는 Gradient Descent Method

장진(Jin Jang) 대한기계학회 2021 대한기계학회 춘추학술대회 Vol.2021 No.4

딥러닝 교육에 대한 다양한 온라인 매체 중 스탠포드대학의 Andrew Un 교수가 온라인 강의플랫폼인 Coursera를 통해 제공하는 딥러닝 강의는 잘 구성되어 있음은 물론이고 쉽고 실용적이어서 많은 타 전공 학생과 기업들이 입문용 강의로 활용하고 있다. 해당 강의를 듣기 위해 필요한 수학 지식 중 비용 함수 최적화(cost function optimization)에서 마주치게 되는 Gradient 개념은 다변수 미적분(multi-variate calculus)에 포함된 내용으로 해당 과목을 수강하지 않은 학습자에게 다소 어려울 수 있으며 전술한 강의에서도 단변수 함수에 대해서는 개념을 설명하지만 다변수 함수에 대해서는 수식만 제공하여 단순히 활용하도록 한다. 하지만 Gradient 개념에 대한 제대로 된 이해와 그 문제점에 대한 고찰이 없으면 Gradient Descent Method에 기초한 다양한 최적화 방법들(GDM, RMSprop, Adam 등)의 필요성과 이해도 어려울 수밖에 없다. 여기에서는 이(2)변수 함수(시각화를 위해)에 대한 Gradient Descent Method를 GeoGebra(무료 수학 소프트웨어)를 이용해 도식적으로 설명함으로써 직관적인 이해가 가능하도록 하였다. 딥러닝을 포함한 다양한 머신 러닝 분야에서 중요한 자리를 차지하고 있는 Gradient의 개념과 최적화에서 Gradient Descent Method의 한계를 이해함으로써 개선된 최적화 방법들을 더 깊이 이해하고 활용하는데 도움이 될 것으로 기대한다. When non-majors learn deep learning, various online contents are available. In particular, deep learning lectures provided by Professor Andrew Un of Stanford University through Coursera, an online lecture platform, are well-structured, easy and practical. The mathematical knowledge required to understand the cost function optimization is the concept of gradient from the multi-variate calculus, which is not easily understood by learners who have not taken the subject, and in the above lecture by Andrew Un, the concept is explained for single variable function only, and for multivariate functions, the formulas are provided. However, without a proper understanding of the concept of gradient, the necessity and understanding of various optimization methods based on gradient descent (GDM, RMSprop, Adam, etc.) will be difficult. Here, GeoGebra (free mathematical software) is used to schematically explain the gradient descent method for a two-variable function for intuitive understanding. It is expected that it will help to understand and utilize modified optimization methods by clearly understanding the concept of gradient of multi-variate functions, which is an important prerequisite knowledge for deep learning, and the limitations of gradient descent.

데이터 예측 모델 최적화를 위한 경사하강법 교육 방법

허경 ( Kyeong Hur ) 한국실천공학교육학회 2022 실천공학교육논문지 Vol.14 No.2

본 논문에서는 기초적인 데이터 예측 모델을 만들고 최적화하는 교육에 초점을 맞추었다. 그리고 데이터 예측 모델을 최적화하는 데 널리 사용되는 머신러닝의 경사하강법 교육 방법을 제안하였다. 미분법을 적용하여 데이터 예측 모델에 필요한 파라미터 값들을 최적화하는 과정에 사용되는 경사하강법의 전체 동작과정을 시각적으로 보여주며, 수학의 미분법이 머신러닝에 효과적으로 사용되는 것을 교육한다. 경사하강법의 전체 동작과정을 시각적으로 설명하기위해, 스프레드시트로 경사하강법 SW를 구현한다. 본 논문에서는 첫번째로, 2변수 경사하강법 교육 방법을 제시하고, 오차 최소제곱법과 비교하여 2변수 데이터 예측모델의 정확도를 검증한다. 두번째로, 3변수 경사하강법 교육 방법을 제시하고, 3변수 데이터 예측모델의 정확도를 검증한다. 이후, 경사하강법 최적화 실습 방향을 제시하고, 비전공자 교육 만족도 결과를 통해, 제안한 경사하강법 교육방법이 갖는 교육 효과를 분석하였다. In this paper, we focused on training to create and optimize a basic data prediction model. And we proposed a gradient descent training method of machine learning that is widely used to optimize data prediction models. It visually shows the entire operation process of gradient descent used in the process of optimizing parameter values required for data prediction models by applying the differential method and teaches the effective use of mathematical differentiation in machine learning. In order to visually explain the entire operation process of gradient descent, we implement gradient descent SW in a spreadsheet. In this paper, first, a two-variable gradient descent training method is presented, and the accuracy of the two-variable data prediction model is verified by comparison with the error least squares method. Second, a three-variable gradient descent training method is presented and the accuracy of a three-variable data prediction model is verified. Afterwards, the direction of the optimization practice for gradient descent was presented, and the educational effect of the proposed gradient descent method was analyzed through the results of satisfaction with education for non-majors.

딥러닝을 위한 경사하강법 비교

강민제 한국산학기술학회 2020 한국산학기술학회논문지 Vol.21 No.2

This paper analyzes the gradient descent method, which is the one most used for learning neural networks. Learning means updating a parameter so the loss function is at its minimum. The loss function quantifies the difference between actual and predicted values. The gradient descent method uses the slope of the loss function to update the parameter to minimize error, and is currently used in libraries that provide the best deep learning algorithms. However, these algorithms are provided in the form of a black box, making it difficult to identify the advantages and disadvantages of various gradient descent methods. This paper analyzes the characteristics of the stochastic gradient descent method, the momentum method, the AdaGrad method, and the Adadelta method, which are currently used gradient descent methods. The experimental data used a modified National Institute of Standards and Technology (MNIST) data set that is widely used to verify neural networks. The hidden layer consists of two layers: the first with 500 neurons, and the second with 300. The activation function of the output layer is the softmax function, and the rectified linear unit function is used for the remaining input and hidden layers. The loss function uses cross-entropy error. 본 논문에서는 신경망을 학습하는 데 가장 많이 사용되고 있는 경사하강법에 대해 분석하였다. 학습이란 손실함수가 최소값이 되도록 매개변수를 갱신하는 것이다. 손실함수는 실제값과 예측값의 차이를 수치화 해주는 함수이다. 경사하강법은 오차가 최소화되도록 매개변수를 갱신하는데 손실함수의 기울기를 사용하는 것으로 현재 최고의 딥러닝 학습알고리즘을 제공하는 라이브러리에서 사용되고 있다. 그러나 이 알고리즘들은 블랙박스형태로 제공되고 있어서 다양한 경사하강법들의 장단점을 파악하는 것이 쉽지 않다. 경사하강법에서 현재 대표적으로 사용되고 있는 확률적 경사하강법(Stochastic Gradient Descent method), 모멘텀법(Momentum method), AdaGrad법 그리고 Adadelta법의 특성에 대하여 분석하였다. 실험 데이터는 신경망을 검증하는 데 널리 사용되는 MNIST 데이터 셋을 사용하였다. 은닉층은 2개의 층으로 첫 번째 층은 500개 그리고 두 번째 층은 300개의 뉴런으로 구성하였다. 출력 층의 활성화함수는 소프트맥스함수이고 나머지 입력 층과 은닉 층의 활성화함수는 ReLu함수를 사용하였다. 그리고 손실함수는 교차 엔트로피 오차를 사용하였다.

두 이종 혼합 모형에서의 수정된 경사 하강법

문상준,전종준 한국데이터정보과학회 2017 한국데이터정보과학회지 Vol.28 No.6

The online learning is a process of obtaining the solution for a given objective function where the data is accumulated in real time or in batch units. The stochastic gradient descent method is one of the most widely used for the online learning. This method is not only easy to implement, but also has good properties of the solution under the assumption that the generating model of data is homogeneous. However, the stochastic gradient method could severely mislead the online-learning when the homogeneity is actually violated. We assume that there are two heterogeneous generating models in the observation, and propose the a new stochastic gradient method that mitigate the problem of the heterogeneous models. We introduce a robust mini-batch optimization method using statistical tests and investigate the convergence radius of the solution in the proposed method. Moreover, the theoretical results are confirmed by the numerical simulations. 온라인 학습은 자료가 실시간으로 혹은 배치 단위로 축적되는 상황에서 주어진 목적함수의 해를 계산하는 방법을 말한다. 온라인 학습 알고리즘 중 배치를 이용한 확률적 경사 하강법 (stochastic gradient decent method) 은 가장 많이 사용되는 방법 중 하나다. 이 방법은 구현이 쉬울 뿐만 아니라 자료가 동질적인 분포를 따른다는 가정 하에서 그 해의 성질이 잘 연구되어 있다. 하지만 자료에 특이값이 있거나 임의의 배치가 확률적으로 이질적 성질을 가질 때, 확률적 경사 하강법이 주는 해는 큰 편이를 가질 수 있다. 본 연구에서는 이러한 비정상 배치 (abnormal batch) 있는 자료 하에서 효과적으로 온라인 학습을 수행할 수 있는 수정된 경사 하강 알고리즘 (modified gradient decent algorithm)을 제안하고, 그 알고리즘을 통해 계산된 해의 수렴성을 밝혔다. 뿐만 아니라 간단한 모의 실험을 통해 제안한 방법의 이론적 성질을 실증하였다.

파라미터 추정을 위한 민감도 기법의 응용에 관한 연구

백문열 韓國工作機械學會 2000 한국생산제조학회지 Vol.9 No.2

This paper deals with the application of sensitivity method to the parameter estimation for the dynamic analysis of gener-al mechanical systems. In this procedure, we take the derivatives of the given system with respect to a certain parameter and use this information to implement the steepest-descent method. This paper will give two examples of this technique applied to simple vehicle models. Simulation results show excellent convergence and accuracy of parameter estimates.

Improved Gradient-descent-based Control Algorithms for Multi-agent Systems With Fixed and Switching Topology

Jiahang Lu,Xiuying Li 제어·로봇·시스템학회 2023 International Journal of Control, Automation, and Vol.21 No.11

In this paper, model free adaptive control algorithms are proposed based on ten improved gradient descent methods which are commonly used as optimization algorithms in deep learning. For the designed control scheme, the modelling, control and optimization can be integrated in a unified framework. The effects of ten algorithms on the consensus tracking performance in multi-agent systems are studied and compared. In order to get a more universal conclusion, systems with fixed and switching topology are considered respectively. Simulation results show that the model free adaptive control algorithm based on adaptive momentum estimation method with decoupled weight decay (AdamW) has optimal performance.

Air Pollution Reduction Based on Intelligent Nonlinear Control Methodology

Yaser Barmayeh,Mehran Afshar,Mohammad Sadegh Dahideh,Mohammad Najafi,Ali Reza Zarei 보안공학연구지원센터 2015 International Journal of Hybrid Information Techno Vol.8 No.1

This paper expands a Multi Input Multi Output (MIMO) fuzzy baseline control (FBC) which controller coefficient is off-line tuned by gradient descent algorithm. The main goal is to adjust the optimal value for fuel ratio (FR) in motor engine. The fuzzy inference system in proposed methodology is works based on Mamdani-Lyapunov fuzzy inference system (FIS). To reduce dependence on the gain updating factor coefficients of the fuzzy methodology, PID baseline method is introduced. This new method provides an optimal setting for other factors which created by PID baseline method. The gradient descent methodology is off-line tune all coefficients of baseline fuzzy based on mathematical optimization methodology. Simulation results signify good performance of fuel ratio in presence of different torque load and external disturbance.

Selecting Fuzzy Rules for Pattern Classification Systems

Lee, Sang-Bum,Lee, Sung-joo,Lee, Mai-Rey Korean Institute of Intelligent Systems 2002 INTERNATIONAL JOURNAL of FUZZY LOGIC and INTELLIGE Vol.2 No.2

This paper proposes a GA and Gradient Descent Method-based method for choosing an appropriate set of fuzzy rules for classification problems. The aim of the proposed method is to fond a minimum set of fuzzy rules that can correctly classify all training patterns. The number of inference rules and the shapes of the membership functions in the antecedent part of the fuzzy rules are determined by the genetic algorithms. The real numbers in the consequent parts of the fuzzy rules are obtained through the use of the descent method. A fitness function is used to maximize the number of correctly classified patterns, and to minimize the number of fuzzy rules. A solution obtained by the genetic algorithm is a set of fuzzy rules, and its fitness is determined by the two objectives, in a combinatorial optimization problem. In order to demonstrate the effectiveness of the proposed method, computer simulation results are shown.

Fuzzy Learning Method Using Genetic Algorithms

Choi, Sangho,Cho, Kyung-Dal,Park, Sa-Joon,Lee, Malrey,Kim, Kitae Korea Multimedia Society 2004 멀티미디어학회논문지 Vol.7 No.6

This paper proposes a GA and GDM-based method for removing unnecessary rules and generating relevant rules from the fuzzy rules corresponding to several fuzzy partitions. The aim of proposed method is to find a minimum set of fuzzy rules that can correctly classify all the training patterns. When the fine fuzzy partition is used with conventional methods, the number of fuzzy rules has been enormous and the performance of fuzzy inference system became low. This paper presents the application of GA as a means of finding optimal solutions over fuzzy partitions. In each rule, the antecedent part is made up the membership functions of a fuzzy set, and the consequent part is made up of a real number. The membership functions and the number of fuzzy inference rules are tuned by means of the GA, while the real numbers in the consequent parts of the rules are tuned by means of the gradient descent method. It is shown that the proposed method has improved than the performance of conventional method in formulating and solving a combinatorial optimization problem that has two objectives: to maximize the number of correctly classified patterns and to minimize the number of fuzzy rules.

Design of Fuzzy-Sliding Model Control with the Self Tuning Fuzzy Inference Based on Genetic Algorithm and Its Application

Go, Seok-Jo,Lee, Min-Cheol,Park, Min-Kyn Institute of Control 2001 Transaction on control, automation and systems eng Vol.3 No.1

This paper proposes a self tuning fuzzy inference method by the genetic algorithm in the fuzzy-sliding mode control for a robot. Using this method, the number of inference rules and the shape of membership functions are optimized without an expert in robotics. The fuzzy outputs of the consequent part are updated by the gradient descent method. And, it is guaranteed that he selected solution become the global optimal solution by optimizing the Akaikes information criterion expressing the quality of the inference rules. The trajectory tracking simulation and experiment of the polishing robot show that the optimal fuzzy inference rules are automatically selected by the genetic algorithm and the proposed fuzzy-sliding mode controller provides reliable tracking performance during the polishing process.