처리량 상한 정규화를 적용한 TCP 혼잡 제어 딥 강화학습 = Deep Reinforcement Learning for TCP Congestion Control with Throughput Cap Normalization

다국어 초록 (Multilingual Abstract)

This study addresses the instability of reinforcement learning (RL)–based Transmission Control Protocol (TCP) congestion control caused by reward-scale drift across bandwidth/round-trip time (RTT) regimes, and proposes Pacemaker-cc, a ceiling-normalized learning approach that estimates a throughput ceiling to normalize per-interval rewards.
The design couples (i) TCP-friendly multiplicative congestion window (cwnd) control at a 0.2s monitor interval (MI), (ii) a dual exponentially weighted moving average (EWMA) tracker for (fast-to-rise, slow-to-fall) updated every second with near-ceiling acknowledgment (ACK)-side bumps, and (iii) immediate safety hooks on retransmission timeout (RTO) and three duplicate acknowledgments (3×dupACK).
In the ns-3 network simulator (ns-3) with ns3-gym, isolating reward normalization improves link utilization from 0.624 to 0.815 and reduces 95th-percentile (p95) RTT inflation from 52.629% to 37.488%. Against alternative ceiling mechanisms—deep Q-network (DQN)-based TCP congestion control algorithm v2 and a Bottleneck Bandwidth and round-trip propagation time (BBR)-like peak estimator—Pacemaker-cc achieves 0.90–1.00 utilization with 2–26% p95 RTT inflation on most capacity–RTT cells, with only a mild weakness at low-RTT/high-capacity. We apply logarithmic cap normalization for input scaling, which preserves mid-range resolution and improves robustness under capacity shifts compared to linear cap (Min–Max) scaling.
Contributions are: (1) a quantitative case for normalization in throughput-driven rewards, (2) a reproducible ns-3 RL-TCP framework, and (3) a simple MI-gated interface with event hooks that “fills the link, not the queue,” yielding stable behavior across regimes. Future work includes fairness under mixed traffic and robustness on cellular/low Earth orbit (LEO) paths.

목차 (Table of Contents)

• List of Tables ⅲ
• List of Figures ⅲ
• List of Abbreviations ⅳ
• Abstract ⅴ
• Ⅰ. Introduction 1
• 1.1 Deep Reinforcement Learning: Context and Motivation 1
• 1.2 TCP Congestion Control 1
• 1.3 Motivation: Why Ceiling-Normalized Learning 4
• 1.4 Our Contributions 4
• Ⅱ. RL Control Interface 9
• 2.1 RL Inputs and Outputs 9
• 2.2 Absolute vs. Relative Signals and scaling 9
• 2.3 Action Space MI Gated and TCP Friendly 10
• 2.4 Reward Throughput Normalized and Hinge shaped 10
• Ⅲ. Pacemaker CC Architecture and Algorithms 12
• 3.1 Decision Workflow Flowchart 12
• 3.2 Implementation Contract 12
• 3.3 Throughput Ceiling Tracker Tmax Periodic Update Rules 15
• 3.4 Ceiling Tracking with Asymmetry 15
• 3.5 ACK-Time Handlers: RTO & Fast Retransmit 16
• 3.6 Master Orchestrator 18
• Ⅳ. Experimental Methodology and Core Results 21
• 4.1. Training and Simulation Setup 21
• 4.2. Reward Normalization and Tmax Baselines 22
• 4.2.1 stage Ⅰ: Normalization-Only 23
• 4.2.2 stage Ⅱ: Throughput Ceiling Baselines & Grid
• Comparison 23
• 4.3 Generalization Across Capacities: LogNorm vs. LinNorm 26
• Ⅴ. Ablation Study 32
• 5.1 Loss Guard & Latency Term Ablation 32
• 5.2 Anchor & Decay Ablation 36
• Ⅵ. Related Work 42
• Ⅶ. Conclusion 45
• Bibliography 46
• ABSTRACT 52