딥러닝을 위한 파이프라인 방식 확률적 경사 하강법 = Pipelined stochastic gradient descent for training deep neural network models

다국어 초록 (Multilingual Abstract)

Parallel train algorithms for deep neural networks (DNNs) are needed to train substantial data. Deep learning has been rapidly growing since 2006 after the introduction of deep belief nets, which DNNs are initialized by the restricted Boltzmann machine. Deep learning has performed well in a variety of classification problems. Many deep learning applications typically perform better with more data. It takes a lot of time for DNN to train a large data set. As data become large, faster train method is needed.
Many parallel learning algorithms are introducing various approximation to speed up. Stochastic gradient descent (SGD) is the most widely used method for training DNNs. Since SGD is an inherently sequential process, the parallelization of SGD is difficult. Delayed gradient problems occur while sequential processes are parallelized.
To avoid the problem of gradient mismatch due to delayed gradients, we improve Pipelined SGD by storing model parameters of each module regarding to the time delay.
The proposed method showed the speedup of x2.25 using 4-GPU without significant performance degradation for Cifar10 dataset.

목차 (Table of Contents)

• 1. Introduction 1
• 2. Background 3
• 2.1 Backpropagation (BP) 3
• 2.2 Data Parallelism 5
• 2.2.1 Synchronous SGD (SSGD) 6
• 2.2.2 Elastic Averaging SGD (EASGD) 7
• 2.2.3 Asynchronous SGD (ASGD) 9
• 2.3 Model Parallelism 10
• 2.3.1 Pipelined SGD 11
• 3. 병렬 알고리즘 복잡도 비교 13
• 3.1 complexity 비교 13
• 3.1.1 Backpropagation complexity 13
• 3.1.2 Parallel algorithms 분석 14
• 4 Proposed Model 16
• 4.1 Pipelined SGD 개선 16
• 5. Experiment 18
• 5.1 MNIST 실험 조건 18
• 5.2 MNIST 학습 속도 18
• 5.3 MNIST 정확도 20
• 5.4 Cifar10 실험 조건 21
• 5.5 Cifar10 실험 결과 21
• 6. 결론 및 향후 과제 24
• 7. References 25
• Appendix 29
• Appendix A. backpropagation의 복잡도 29