RCAF: A Multimodal Adaptive Fusion Framework for 3D Object Detection under Sensor Corruption

다국어 초록 (Multilingual Abstract)

Autonomous vehicles require robust multimodal perception systems to oper- ate reliably under adverse weather conditions and sensor degradation. This thesis presents RCAF, a confidence-aware Radar–Camera Adaptive Fusion framework for resilient 3D object detection under sensor corruption. The proposed frame- work refines radar features through Gaussian spreading and dynamically balances camera and radar features in bird’s-eye view （BEV） space using an Adaptive Confidence Score mechanism. Unlike conventional fusion approaches that employ fixed fusion weights, RCAF learns confidence scores in an end-to-end manner, en- abling context-aware fusion that prioritizes radar information under low-visibility conditions and camera information in clear environments. Experimental evalua- tion on the nuScenes dataset, the nuScenes-Corrupt benchmark, and under radar noise corruption demonstrates that RCAF achieves competitive performance rel- ative to state-of-the-art baselines while maintaining strong robustness across di- verse camera and radar corruption scenarios. The results highlight the effective- ness of adaptive confidence-based fusion for enhancing multimodal perception robustness and demonstrate the potential of RCAF as a resilient framework for safety-critical autonomous driving applications.

목차 (Table of Contents)

• 1 Introduction 1
• 1.1 Motivation 1
• 1.2 Problem Statement 5
• 1.3 Objective of the Thesis 6
• 1.4 Organization and Contributions 7
• 2 Background and Related Work 9
• 2.1 Background 10
• 2.1.1 Autonomous Driving Perception 10
• 2.1.2 Bird’s-Eye View (BEV) Representation 11
• 2.2 Related Work 12
• vi
• 2.2.1 Radar-only Perception 13
• 2.2.2 Camera-only Perception 14
• 2.2.3 Radar–Camera Fusion 16
• 2.2.4 Summary 18
• 3 Confidence-Aware Radar–Camera Adaptive Fusion Framework 19
• 3.1 Introduction 19
• 3.2 Overall Architecture of RCAF 20
• 3.3 Camera Feature Extraction 22
• 3.4 Density-Guided Gaussian Spreading 23
• 3.5 Radar Feature Enhancement via Joint and Independent Fusion 24
• 3.6 Adaptive Confidence-Aware Fusion 25
• 3.6.1 Modality Availability Gating 26
• 3.6.2 Adaptive Confidence Learning 26
• 3.7 Loss Function and Optimization 28
• 3.8 Summary 29
• 4 Experimental Setup and Evaluation Protocol 30
• 4.1 Introduction 30
• 4.2 Datasets 31
• 4.3 Sensor Corruption Benchmarks 31
• 4.3.1 Camera Corruptions 33
• 4.3.2 Radar Noise Corruptions 33
• 4.4 Training and Implementation Details 35
• 4.5 Evaluation Metrics 36
• vii
• 4.5.1 Mean Average Precision (mAP) 37
• 4.5.2 nuScenes Detection Score (NDS) 37
• 4.5.3 True Positive Error Metrics 37
• 4.6 Summary 39
• 5 Results and Discussion 40
• 5.1 Introduction 40
• 5.2 Performance on the nuScenes Validation Set 41
• 5.3 Robustness to Radar Noise with Clean Camera Inputs 42
• 5.4 Robustness under Camera Corruption (nuScenes-C) 43
• 5.4.1 Corruption-Specific Analysis 44
• 5.5 Robustness under Combined Radar Noise and Camera Corruption 46
• 5.6 Discussion 47
• 5.7 Summary 48
• 6 Conclusion 50
• 6.1 Concluding Remarks 50
• 6.2 Limitations 51
• 6.3 Future Directions 52
• Bibliography 53
• 요약 59
• viii