컨테이너 터미널 운영에서 AGV 작업 배정을 위한 ACO와 딥 Q-러닝 모델의 비교 = The comparison of ACO and deep Q-learning models for AGV dispatching in container terminal operations

다국어 초록 (Multilingual Abstract)

This study addresses the integrated scheduling problem of berth planning, quay crane operations, and automated guided vehicle task assignment in automated container terminals. As container terminals become increasingly automated and complex, the coordination between quay cranes and AGVs has become a critical factor affecting terminal productivity, equipment utilization, and vessel turnaround time. In particular, inefficient AGV task assignment may cause quay crane waiting, AGV idle time, unnecessary travel, and delays in the overall handling process. Therefore, an effective scheduling approach is required to minimize the total completion time of container handling operations.
To solve this problem, this study formulates a Berth/QC–AGV integrated scheduling environment that considers vessel berthing positions and times, quay crane task allocation, AGV dispatching, travel time, battery status, and charging constraints. Based on this environment, two solution approaches are compared: Ant Colony Optimization （ACO）, a metaheuristic algorithm suitable for combinatorial optimization problems, and Rainbow-DQN, a deep reinforcement learning model that integrates several improved DQN techniques. The objective of both models is to minimize makespan, which represents the completion time of all container handling and transportation tasks.
Computational experiments were conducted under various combinations of container volumes and AGV fleet sizes. The experimental results show that Rainbow-DQN generally outperformed ACO in terms of makespan. Among 72 experimental conditions, Rainbow-DQN achieved better results in 69 cases, while ACO showed better performance in only 3 cases. The performance improvement of Rainbow-DQN was especially noticeable under conditions where AGV assignment decisions had a strong influence on overall scheduling efficiency. In addition, while ACO required repeated search processes for each experimental condition, Rainbow-DQN required an initial training process but was able to generate scheduling decisions rapidly through policy inference after training.
The results indicate that ACO can provide feasible and practical solutions for complex AGV scheduling problems, but its computational burden increases as the problem size grows. In contrast, Rainbow-DQN shows strong potential for large-scale and dynamic scheduling environments by learning effective task assignment policies from repeated interactions with the scheduling environment. Therefore, this study suggests that deep reinforcement learning can be an effective alternative for AGV task assignment and integrated scheduling in automated container terminal operations.

목차 (Table of Contents)

• I. 서 론 1
• 1. 연구배경 1
• 2. 배경 이론 및 연구 현황 5
• 1） 컨테이너 터미널에서의 안벽 크레인과 AGV 시스템 5
• 2） Ant Colony Optimization（ACO） 7
• 3） 강화학습 기반 스케줄링과 Rainbow-DQN 8
• II. 실험 문제 구성 및 설계 12
• 1. 문제 환경 및 구조 12
• 2. 수리 모형 14
• 1） Berth/QC 스케줄링 수리 모형 14
• 2） AGV 스케줄링 수리 모형 19
• III. 개미 군집 최적화（ACO） 29
• 1. 개미 군집 최적화 （ACO） 알고리즘 29
• 1） ACO 기반 통합 항만 운영 스케줄링 모형 29
• 2） ACO 알고리즘의 개념 29
• 3） ACO 알고리즘의 특징 30
• 4） ACO 알고리즘의 기본 구성 요소 31
• 5） ACO 알고리즘의 기본 절차 33
• 6） ACO 알고리즘을 기반으로 한 Berth/QC-AGV 스케줄링 의사코드 35
• 7） ACO 알고리즘의 장점 36
• 8） ACO 알고리즘의 표현 37
• IV. Rainbow-DQN 40
• 1. Rainbow-DQN 모델 40
• 1） Rainbow-DQN의 정의 40
• 2） Rainbow-DQN의 배경 41
• 3） Rainbow-DQN의 구성 요소 41
• 4） Rainbow-DQN의 통합 구조 45
• 5） Rainbow-DQN의 Distributional n-step Target 46
• 6） Rainbow-DQN의 손실함수 47
• 7） Rainbow-DQN을 기반으로 한 Berth/QC-AGV 스케줄링 의사코드 48
• 8） Rainbow-DQN의 특징 50
• 9） Rainbow-DQN의 장점 51
• V. 실험 결과 및 분석 53
• 1. 실험 환경 및 파라미터 설정 53
• 1） 실험 환경 53
• 2） ACO 기반 스케줄링 파라미터 설정 53
• 3） Rainbow-DQN 기반 스케줄링 파라미터 설정 56
• 2. 실험 결과 58
• 1） ACO 기반 Berth/QC-AGV 스케줄링 결과 58
• 2） Rainbow-DQN 기반 Berth/QC-AGV 스케줄링 결과 62
• 3. 결과 비교 및 분석 65
• 1） ACO와 Rainbow-DQN의 결과 비교 65
• 2） ACO와 Rainbow-DQN의 계산시간 비교 67
• 3） 알고리즘별 성능 차이 분석 70
• 4） 컨테이너 수 증가에 따른 비교 분석 72
• 5） AGV 수 증가에 따른 비교 분석 73
• VI. 결론 74
• 참고문헌 75
• ABSTRACT 80