Proactive congestion avoidance for distributed deep learning

다국어 초록 (Multilingual Abstract)

This thesis presents “Proactive Congestion Notification” (PCN), a congestionavoidance technique for distributed deep learning (DDL). DDL is widely used to scale out and accelerate deep neural network training. In DDL, each worker trains a copy of the deep learning model with different training inputs and synchronizes the model gradients at the end of each iteration. However, it is well known that the network communication for synchronizing model parameters is the main bottleneck in DDL. Our key observation is that the DDL architecture makes each worker generate burst traffic every iteration, which causes network congestion and in turn degrades the throughput of DDL traffic. Based on this observation, the key idea behind PCN is to prevent potential congestion by proactively regulating the switch queue length before DDL burst traffic arrives at the switch, which prepares the switches for handling incoming DDL bursts. In our evaluation, PCN improves the throughput of DDL traffic by 72% on average.

목차 (Table of Contents)

• 1 Introduction 1
• 2 Background and Motivation 6
• 2.1 Distributed Deep Learning Traffic 6
• 2.2 Related Work 10
• 2.2.1 Host-side scheduling on DDL traffic 10
• 2.2.2 Reducing the amount of DDL Traffic 12
• 2.2.3 Improvement using an in-network switch 13
• 2.2.4 ECN-based approach 15
• 2.2.5 Novelty of PCN 16
• 2.3 P4 and Switch Programmability 17
• 3 Design 19
• 3.1 Communication Sequence between Worker and PS 19
• 3.2 Switch Operation for PCN 21
• 3.3 PCN Threshold Policy 24
• 4 Evaluation 26
• 4.1 Evaluation Metrics 27
• 4.2 Queue Length Change 28
• 4.3 Performance Improvement 30
• 4.4 Overheads 31
• 5 Discussion 32
• 5.1 Estimation of the total DDL training time 32
• 5.2 The impact of environment changes on PCN 33
• 5.3 Multi-tenancy 34
• 6 Conclusion 35
• Bibliography 36