NN Maestro : scheduling for high-performance inference of complex deep learning models in multi-GPU systems

다국어 초록 (Multilingual Abstract)

Recent complex neural network models, which have shown competitive accuracy, face the challenge of deploying them efficiently across multiple GPUs. Under manual scheduling, it is almost impossible to achieve fair performance gains due to the inter-layer dependencies of the neural network models and the variable overhead of data communication. To solve this problem, we propose NN Maestro, a layer scheduling algorithm for multi-GPU systems that generate efficient parallel execution strategies that minimize data communication overhead, thereby improving inference latency for complex neural networks. In NN Maestro, a pre-trained SVM classifier first determines the benefit of multi-GPU scheduling of target graphs. For each graph decided to be parallelized, a scheduling order is calculated considering its topological order and Significance Cost. In case parallel execution seems profitable, the best GPUs are determined by comparing their Placement Cost. Once the layer mapping to the GPUs has been determined, the final schedule is generated by grouping the layers that can run simultaneously. To verify our work, NN Maestro achieves up to 1.67x performance improvements over the baseline in multi-GPU configurations (2 2080Ti, 4 V100, and 4 A6000 GPUs).

목차 (Table of Contents)

• Chapter 1. Introduction 1
• Chapter 2. Background and Motivation 4
• Chapter 3. NN Maestro 9
• Chapter 4. Experimental Results 22
• Chapter 5. Future Directions 36
• Chapter 6. Related Works 37
• Chapter 7. Conclusion 39
• Reference 40
• Appendix 44