http://chineseinput.net/에서 pinyin(병음)방식으로 중국어를 변환할 수 있습니다.
변환된 중국어를 복사하여 사용하시면 됩니다.
딥러닝 학습에서 동기화 배리어 재배치와 파이프라이닝을 이용한 Double-Averaging 가속
유찬희(Chanhee Yu),박경석(Kyongseok Park) Korean Institute of Information Scientists and Eng 2021 정보과학회논문지 Vol.48 No.11
In deep learning using distributed computing, synchronization is one of the most important factors. While Local SGD is a low-frequency synchronization method that enables fast training, it is limited by high convergence difficulties. And Double-Averaging and SlowMo have been proposed to reduce the convergence difficulties of Local SGD. Double-Averaging improves the convergence difficulties by adding momentum buffer synchronization. However, the training time also increases due to the increased data synchronization. On the other hand, SlowMo adds a Two-layer momentum structure to the Local SGD resulting in reduced convergence difficulties without additional synchronization. However, this requires finding the appropriate SlowMo hyper-parameters. Therefore, in this paper, we proposed accelerated Double-Averaging via synchronization barrier repositioning and pipelining. The proposed method significantly reduced the convergence difficulties and accelerated performance.