Enhancing deep-learning-as-a-service with optimized architectures : a framework focused on service-level-objectives

다국어 초록 (Multilingual Abstract)

As artificial intelligence technology advances, the complexity of AI models increases, and the computing resources required by these models continue to grow. This results in cost issues when scaling AI services.
Therefore, how to efficiently serve AI models has become a critically important problem, leading to research from various perspectives.
Among various studies, research on model inference optimization applicable in on-premises environments has been gaining attention. Model inference optimization focuses on minimizing the latency of AI model services and maximizing the performance of models deployed on servers. To achieve this, it aims to determine the most efficient settings for various inference server parameters (such as max batch size, max queue delay time, model instances, dynamic inference, etc.). By optimally configuring the model service architecture, it is possible to enhance the performance of model inference services using the same models and hardware.
However, there are limitations in configuring the optimal service architecture due to the long exploration time required. Moreover, solely utilizing quality metrics as the criteria for judging service architecture may lead to the problem of absence of objective criteria for selecting optimal configurations.
Therefore, this study proposes a new integrated metric for determining the optimal service architecture and introduces a service architecture search framework to minimize exploration costs.
In this paper, experiments are conducted in real-world service environments, demonstrating the effectiveness of the proposed integrated metric and framework in improving performance.

목차 (Table of Contents)

• 1. Introduction 1
• 1.1 Background 1
• 1.2 Research Scope 3
• 1.3 Research Purpose 6
• 1.4 Outlook of this Thesis 7
• 2. Related Research 8
• 2.1 Background 8
• 2.2 Literature Review 9
• 2.3 Limitations of Previous Research 11
• 3. Proposed Method 14
• 3.1 Modified Desirability Function 14
• 3.2 Service Architecture Search Framework 21
• 4. Experiments 24
• 4.1 Experimental Setting 24
• 4.2 Experimental Results 26
• 5. Conclusion 43
• 5.1 Conclusion 43
• 5.2 Limitations and Future Work 46