A Multi-Agent Deep Reinforcement Learning Framework for Energy-Efficient Application Placement in Federated Cloud?Edge Systems = 다중 에이전트 심층 강화학습을 통한 연합 클라우드?엣지 컴퓨팅의 에너지 효율 최적화

다국어 초록 (Multilingual Abstract)

The fast growth of cloud and edge computing technologies is changing digital services and how data is processed, closer to users. Nonetheless, this development increased the consumption of energy, carbon, and the complexity of resource management. Sustainability and efficiency in such federated environments require intelligent orchestration mechanisms capable of achieving performance, scalability, and sustainability. This doctoral research develops a Multi-Agent Deep Reinforcement Learning-based intelligent decentralized orchestration framework that can optimize energy consumption and sustainability when placing applications in a federated cloud-edge system to address these issues.
The research is divided into two stages. The initial phase involves conducting a Systematic Literature Review (SLR) based on the Kitchenham protocol to examine different types of ML-based energy optimization techniques in the cloud-edge. In order to develop a comprehensive taxonomy of the optimization methods, we carried out a review of 73 peer-reviewed studies. The optimization methods include supervised learning, unsupervised learning, and Reinforcement learning (RL) models. There are three major gaps which the research has identified from the literature review. In the first place, no urgency-aware mechanisms which react dynamically to load variability exist. Second, the optimization objectives do not include environmental and carbon metrics. Thirdly, there is a lack of interoperability and also decentralization across federated infrastructures. Thus, the gaps show that there is need for learning based, multi-objective and distributed orchestration paradigm for enhancing energy efficiency.
The second stage develops a MA-DRL based orchestration framework using MAPPO for discrete decisions on application placement and MADDPG based control for continuous Dynamic Voltage and Frequency Scaling (DVFS) application placement. The organization operates based on a CTDE concept. During training, this enables the agents to learn through cooperation. However, the agents will act independently when deployed. A multi-objective reward model is developed to jointly optimize energy consumption, latency, SLA violations, and carbon cost.
The framework has been implemented in a SimPy–PyTorch simulation environment modelling realistic federated infrastructures of heterogeneous cloud and edge nodes. The experiments show that the proposed MA-DRL model can reduce energy consumption by 25–35%. Also, the latency improves by around 18–22%. In addition, SLA violations and carbon emissions will also significantly reduce the conventional baselines which include Shortest Queue (SQ), Round Robin (RR), Cloud-Only (CO) and Energy-Aware Heuristic (EAH). The framework can also scale up with increased demand and ensure cooperation among agents against the changing workload.
The research impacts the theory because it provides (1) a set of ML-driven methods for energy optimization; (2) a conceptual model to link energy, SLA, and carbon efficiency; and (3) a multi-agent cooperation of sustainable orchestration of federated systems without central coordination. contributions as a whole increase the understanding of how AI can drive sustainability in distributed computing, which is in line with wider efforts to promote low-carbon digital infrastructures and the Sustainable Development Goals (SDG 9: Industry, Innovation and Infrastructure; SDG 13: Climate Action).
To sum up, this thesis demonstrates a fully analytical and experimental framework for smart, efficient and carbon-aware orchestration for federated cloud-edge system. This connects those theories and practice for the future studies of the autonomous green computing. Furthermore, the research offers useful guidance on integrating AI-based orchestration into a current digital ecosystem.

국문 초록 (Abstract)

지금의 클라우드와 엣지컴퓨팅 기술의 빠른 발전은 디지털 서비스와 데이터 처리 방식을 사용자의 곁으로 더욱 가깝게 만들고 있다. 그러나 이러한 발전으로 인해 에너지 소비와 탄소 배출량이 증가하였고, 자원이 더 복잡해졌습니다. 연합 환경 내에서 성능, 확장성 및 지속 가능성을 모두 가질 수 있는 지능형 오케스트레이션 메커니즘이 필요하다. 본 박사 연구는 연합 클라우드-엣지 시스템에서 애플리케이션을 배치할 때 에너지를 소모하고 지속가능성을 최적화할 수 있는 다중 에이전트 심층 강화 학습 기반의 지능형 분산 오케스트레이션 프레임워크를 개발합니다.

이 연구는 두 단계로 이루어져 있다. 첫 번째 단계에서는 키친햄 프로토콜(Kitchenham protocol)을 바탕으로 한 체계적 문헌 고찰(Systematic Literature Review, SLR)기법을 활용하여 클라우드-엣지 환경에서의 머신러닝(ML) 기반 에너지 최적화 기법을 조사하는 것이다. 최적화 방법론의 포괄적인 분류 체계를 개발하기 위하여 73편의 동료 검토 논문을 훑어보았다. 검토된 최적화 기법은 감독학습, 비감독학습, 강화학습(RL)모델입니다. 문헌 검토 도중, 세 가지 주된 연구의 공백이 발견되었다. 부하 인식이 통계적으로 반응하고 있어 동적 전개가 어렵습니다. 둘째, 최적화 목적에 환경 및 탄소 배출 향상 지표가 포함되지 않는 것. 첫 번째, 연합 인프라 전반에 대한 상호 운용성의 부족과 분산의 부족이다. 에너지 효율성 향상을 위한 학습 기반 다목적 분산 오케스트레이션 패러다임이 필요한 이유.

두 번째 단계에서는 애플리케이션 배치의 이산적 결정을 위해 MAPPO와 연속적 DVFS 애플리케이션 배치의 MA-DRL 기반 오케스트레이션 프레임워크를 개발한다. 이 시스템은 CTDE(Computational Team Development Enhancement) 개념을 이용합니다. 학습때는 협력하여 학습하지만 배포때는 혼자 작동. 에너지 소비, 지연 시간, SLA 위반 및 탄소 배출량을 동시에 최적화하기 위한 다목적 보상 모델이 개발되었습니다. SimPy-PyTorch 시뮬레이션 환경은 이기종 클라우드 및 엣지 노드로 구성된 현실적인 연합 인프라를 모델링하다. 실험 결과 Proposed MA-DRL 모델은 다운링크와 업링크에서 각각 25∼35%의 에너지 소비시 지연시간도 약18∼22%에 개선할 수 있음을 보였다. 또한, SLA violations 및 carbon emission과 관련하여 SQ, RR, CO, EAH 를 포함한 기존 기준 모델들보다 크게 향상되었습니다. 수요의 증가에 따라 이 프레임워크는 확장 가능할 것이며, 변화하는 워크로드에 적응하여 에이전트 간의 협력을 보장합니다.

본 연구는 에너지 최적화를 위한 머신러닝 기반 방법론, 에너지 및 서비스수준계약(SLA) 및 탄소 효율성을 연계하는 개념 모델, 중앙조정 없이 연합시스템의 지속가능한 오케스트레이션을 위한 다중에이전트 협력 모델을 제시하여 이론에 중요한 기여를 합니다. 인공지능(AI)이 분산 컴퓨팅 환경에서 지속가능성을 향상시키는 방안을 더욱 잘 이해 할 수 있는 기여 자료로 저탄소 디지털 인프라 및 지속가능발전목표(SDG 9: 산업, 혁신 및 인프라; SDG 13: 기후 행동) 달성을 위한 더 넓은 노력과 함께 놓여 있다.

실질적으로, 논문에서는 분포된 클라우드-엣지 시스템 오케스트레이션의 주요 요소들과 기여도를 제시하고 있습니다. 자율적 친환경 컴퓨팅, 이론과 실천의 교차점으로서의 미래 연구 또한 이 연구는 AI 오케스트레이션의 현 디지털 생태계 통합을 위한 설계참조를 제시한다.

목차 (Table of Contents)

• Abstract iii
• Contents vi
• List of Tables xi
• List of Figures xiii
• List of Abbreviations xvi
• Chapter 1 Introduction 1
• 1.1 Background and Context 1
• 1.2 Problem Statement and Research Gap 2
• 1.3 Motivation and Research Significance 8
• 1.4 Research Objectives and Research Questions 10
• 1.4.1 Study 1: SLR 10
• 1.4.2 Study 2: Experimental Framework 11
• 1.5.Research Positioning 16
• 1.6 Research Hypotheses 16
• 1.7 Research Model Overview 17
• 1.8 Synthesis: Bridge Between Study 1 and Study 2 19
• 1.9 Research Methodology 21
• 1.9.1 Study 1: SLR 21
• 1.9.2 Study 2: Experimental Framework (MA-DRL) 22
• 1.10 Research Outline 23
• Chapter 2 . Background and Conceptual Foundations for Cloud–Edge Orchestration 26
• 2.1 Introduction to Cloud–Edge Orchestration 26
• 2.2 Energy-Efficient Resource Management in Cloud–Edge Systems 28
• 2.3 SLA-Aware and Latency-Constrained Task Placement 31
• 2.4 Sustainability and Carbon-Aware Cloud–Edge Orchestration 33
• 2.5 Federated, Cooperative, and Decentralized Orchestration Approaches 34
• 2.5.1 Centralized Orchestration: Applicability and Limitations 34
• 2.5.2 Federated, Cooperative, and Decentralized Approaches 36
• 2.6 Application Placement in Cloud–Edge Systems 42
• 2.6.1 Heuristic and Rule-Based Placement Strategies 43
• 2.6.2 Optimization-Based Placement Approaches 44
• 2.6.3 Discussion and Limitations of Non-Learning Methods 45
• 2.7 Reinforcement Learning for Cloud–Edge Resource Orchestration 46
• 2.8 Limitations of Existing RL/MARL Solutions 49
• Chapter 3 . Optimizing Energy Efficiency in Cloud-Edge Computing Through ML: A Systematic Literature Review 51
• 3.1 Introduction 51
• 3.2 Related work 55
• 3.2.1 User Applications (SaaS) 56
• 3.2.2 Platform as a Service (PaaS) 58
• 3.2.3 Infrastructure as a Service (IaaS) 60
• 3.2.4 Comparison and Research Gap 64
• 3.3 Methodology 69
• 3.3.1 Study Selection Process 69
• 3.3.2 Overview of the Selected Studies 76
• 3.4 Analysis 81
• 3.4.1 Descriptive Analysis 81
• 3.4.2 Gaps Specific to Each Research Question 84
• 3.5 Discussion 95
• 3.5.1 Comparison with Prior Reviews 96
• 3.5.2 Theoretical Implications 99
• 3.5.3 Practical Implications and Unified Research Perspective 101
• 3.5.4 Research Agenda and Open Challenges 104
• 3.5.5 Limitation 109
• 3.6. Synthesis of Study 1 Findings 111
• 3.7.Conclusion 112
• Chapter 4 . Framework for Energy-Efficient Application Placement in Federated Cloud-Edge Systems 115
• 4.1. Introduction 115
• 4.1.1 Background 115
• 4.1.2. Motivation and problem description 116
• 4.1.3. Research objectives and Research question 118
• 4.1.4 Relevance of the study 119
• 4.2 Related work 120
• 4.2.1 Comparative Review of related works 120
• 4.2.2 Synthesis and Key Observations 129
• 4.2.3 Research Gap in Existing Literature 130
• 4.2.4 Research Questions 132
• 4.3 Architectural Framework and Algorithm 135
• 4.3.1 Real world scenario Illustration 135
• 4.3.2 Proposed Hybrid MA-DRL Framework 144
• 4.3.3 Formal MAMDP Formulation of the Proposed Framework 147
• 4.3.4 Hybrid DRL Algorithms Under CTDE 153
• 4.4 Simulation Setup and Design 166
• 4.4.1. Simulation Environment 167
• 4.4.2 Simulation Parameters and Input Datasets 171
• 4.4.3. Learning and Deployment Workflow 176
• 4.4.4. Dataset and Observation Structure 179
• 4.4.5 Evaluation Metrics and Baselines 189
• 4.5 Experimental Results and Analysis 194
• 4.5.1. Training Environment and Configuration Parameters 194
• 4.5.2. Training Performance 204
• 4.5.3. Energy Efficiency Evaluation 215
• 4.5.4 Overhead of Learning Agents and Net Energy Savings 220
• 4.5.5 Latency and QoS/SLA Performance 222
• 4.5.6 Environmental Sustainability 231
• 4.5.7 Scalability and Agent Negotiation 233
• 4.5.8 Robustness Check 237
• 4.5.9 Sensitivity Analysis 245
• 4.5.10 Ablation Studies 256
• 4.5.11 Comparative Summary 279
• 4.6 Discussion and implications 291
• 4.6.1 Theoretical Implications 291
• 4.6.2 Practical Implications 292
• 4.6.3 Methodological Implications 293
• 4.6.4 Research and Policy Implications 293
• 4.6.5 Summary of Key Insights 294
• Chapter 5 . Conclusion 295
• 5.1 Summary of the Work 295
• 5.2 Key Findings and Contributions 298
• 5.3 Discussion of Results 300
• 5.4 Limitations of the Study 303
• 5.5 Future Work 305
• 5.6 Concluding Remarks 306
• Bibliography 309
• Appendix 327
• Appendix 1: Key words and Boolean search term combinations 327
• Appendix 2: Summary of Contributions of Papers 329
• Appendix 3: Overall Distribution of the 73 Selected Studies 341
• Appendix 4: Overall Distribution of the 73 Selected Studies 341
• Appendix 5: Training Hyperparameters of the Framework 342
• Appendix 6: Algorithmic Details of the Hybrid MA-DRL Framework 343
• 6.1 Centralized Training and Decentralized Execution (CTDE) 343
• 6.2 MAPPO for Discrete Application Placement 343
• 6.3 MADDPG for Continuous Resource Control 345
• 6.4 Experience Collection and Replay 346
• 6.5 Hybrid Policy Execution 346
• 6.6 Remarks on Stability and Scalability 347
• Acknowledgments 348
• Abstract (Korean) 349