Semantic domain knowledge-based localization with multi-sensor fusion and hierarchical representation

다국어 초록 (Multilingual Abstract)

As autonomous driving systems advance toward Level 4 autonomy, localization becomes a safety-critical component that directly affects planning, decision-making, and vehicle control. Real-world driving environments, however, present frequent GNSS outages, sensor degradation, environmental changes, and hardware faults. Conventional localization methods, which rely primarily on geometric consistency and numerical optimization, struggle to maintain reliability under such conditions. In particular, purely geometric approaches often accept physically or contextually implausible pose hypotheses when they minimize numerical error, leading to accumulated drift and context-inconsistent localization during long-term operation.
This dissertation addresses these limitations by redefining vehicle localization as a context-aware constraint selection problem rather than a purely geometric optimization task. The proposed framework integrates ontology-based semantic modeling and reasoning directly into the localization process and employs behavior-aware reasoning to select semantic constraints that align with the current driving context. By evaluating logical and contextual validity at the constraint generation stage, the framework prevents inconsistent or misleading information from entering the estimation process, even under sensor faults or transient observation anomalies.
To integrate semantic reasoning while meeting real-time requirements, this work introduces a hierarchical representation with a modular architecture. The framework organizes sensor information into three levels: low-level geometric features for real-time motion estimation, object-level semantic representations for constraint generation and sensor fusion, and scene-level structural context for higher-level semantic reasoning. Each level operates as an independent module, enabling a clear separation between time-critical perception and higher-level reasoning. This design allows the system to selectively exploit semantically relevant sensor information while maintaining computational efficiency.
Extensive experiments across diverse driving scenarios and sensor degradation conditions demonstrate that the proposed semantic-aware localization framework achieves greater stability and robustness than conventional geometric approaches. Furthermore, the object-centric semantic representation naturally supports cooperative perception by enabling consistent information sharing among distributed agents. Overall, this work establishes semantic reasoning as a core component of localization and presents a practical framework that enhances reliability, robustness, and extensibility for real-world autonomous driving systems.

국문 초록 (Abstract)

자율주행 기술이 레벨 4 단계로 진입함에 따라, 위치 인식은 안전성과 직결되는 핵심 요소로 자리 잡고 있다. 그러나 실제 주행 환경에서는 GNSS 장애, 센서 성능 저하, 환경 변화, 하드웨어 고장 등이 빈번하게 발생하며, 기존의 기하적 정합 중심 로컬라이제이션 방식은 이러한 조건에서 일관성과 신뢰성을 유지하는 데 한계를 보인다. 특히 물리적·문맥적으로 타당하지 않은 위치 가설이 기하적 오차 최소화만으로 선택되는 문제는 장기 운용 시 누적 오차와 안전성 저하로 이어진다.

본 논문은 이러한 한계를 극복하기 위해, 로컬라이제이션을 단순한 수치 최적화 문제가 아닌 문맥 인식 기반 제약 선택 문제로 재정의한다. 이를 위해 온톨로지 기반 시맨틱 모델링과 추론을 로컬라이제이션 단계에 통합하고, 주행 상황을 해석하는 행동 추론 메커니즘을 통해 현재 맥락에 부합하는 시맨틱 제약만을 선택적으로 적용하는 프레임워크를 제안한다. 이 접근법은 논리적·의미적으로 부적절한 위치 가설을 제약 생성 단계에서 사전에 배제함으로써, 센서 오류나 일시적 관측 이상 상황에서도 일관된 위치 추정을 가능하게 한다.

또한 본 논문은 실시간성과 확장성을 동시에 만족하기 위해, 저수준 기하 정보, 객체 수준 시맨틱 정보, 장면 수준 구조 정보를 포함하는 계층적 표현 구조와 모듈형 아키텍처를 제안한다. 각 계층은 독립적으로 동작하며, 하위 계층은 실시간 이동 추정을 담당하고 상위 계층은 의미적 제약과 문맥 추론을 수행한다. 이러한 구조는 이종 센서로부터 수집되는 방대한 정보를 주행 상황과 의미적으로 관련된 정보로 선별하여 활용함으로써, 효율적이고 구조화된 다중 센서 융합을 가능하게 한다.

다양한 실환경 주행 시나리오와 센서 고장 조건에서 수행한 실험 결과는, 제안한 시맨틱 기반 로컬라이제이션 프레임워크가 기존 기하 중심 방법 대비 향상된 안정성과 강인성을 제공함을 보여준다. 더 나아가 객체 중심 시맨틱 표현을 활용한 구조는 차량 간 정보 공유를 통한 협력 인식으로 자연스럽게 확장될 수 있음을 확인하였다. 본 연구는 시맨틱 정보를 로컬라이제이션의 보조 요소가 아닌 핵심 제약으로 통합함으로써, 실제 자율주행 환경에서 요구되는 신뢰성과 확장성을 동시에 만족하는 실용적인 로컬라이제이션 프레임워크를 제시한다.

목차 (Table of Contents)

• Chapter 1. Introduction 1
• Chapter 2. Related Works 5
• 2.1 Traditional Localization and SLAM 5
• 2.2 Standardization and Evolution of HD Mapping Techniques 6
• 2.3 Multi-Sensor Fusion-Based Localization 7
• 2.4 Learning-Based Localization and Place Recognition 8
• 2.5 Semantic Mapping and Semantic SLAM 9
• 2.6 Semantic Reasoning and Planning-Oriented Approaches 10
• 2.7 Summary and Motivation 11
• Chapter 3. Semantic Modeling Framework 12
• 3.1 Semantic Knowledge Representation 13
• 3.1.1 Basic Components of HD Map 13
• 3.2 Semantic HD Map 16
• 3.2.1 Semantic Object Selection Criteria 18
• 3.2.2 Environmental Elements and Semantic Definitions Based on TOSM 19
• 3.2.3 Object Properties for Spatial and Robot Relations 20
• 3.2.4 Datatype Properties in the Triplet Ontological Semantic Model 23
• 3.3 Semantic Reasoning for Contextual Consistency 25
• 3.4 Semantic Domain Modeling for Localization 28
• 3.4.1 Semantic Type Hierarchy for Localization 29
• 3.4.2 Spatial Connectivity 30
• 3.4.3 Vehicle State Representation. 30
• 3.4.4 Sensor-Level Observation Predicates. 31
• 3.4.5 HD Map-Derived Semantic Constraints. 32
• 3.4.6 Localization Reasoning Actions. 33
• 3.4.7 Semantic Localization Problem Definition 38
• 3.4.8 Semantic Problem Generation from On-Demand Database 38
• 3.4.9 Semantic Reasoning-Driven Localization Constraints 39
• Chapter 4. Hierarchical Scene Representation 41
• 4.1 Hierarchical Image Representation Network 42
• 4.1.1 Overview of Network Architecture 42
• 4.1.2 Low-Level Feature Abstraction 43
• 4.1.3 Object-Level Representation 44
• 4.1.4 Descriptor Modules 45
• 4.1.5 Center Pooling and Graph Construction 46
• 4.1.6 Shape Descriptor 48
• 4.1.7 Color Descriptor 50
• 4.2 Scene-Level Semantic Embedding 52
• 4.2.1 Descriptor Fusion and Topology Embedding 53
• 4.2.2 Loss Functions and Training Objectives 55
• 4.3 Hierarchical Point-cloud Representation 57
• 4.4 Multi-Sensor Fusion via Hierarchical Representation 57
• Chapter 5. Localization with Hierarchical Scene Understanding 59
• 5.1 System Overview 59
• 5.1.1 Dual-Loop Architecture for Real-Time and Semantic-Aware Localization 60
• 5.1.2 Distributed Processing Architecture for Real-time 62
• 5.2 Background: Motion Models and Sensor Models 63
• 5.2.1 Vehicle Kinematic and Dynamic Models 64
• 5.2.2 IMU Motion and Bias Model 65
• 5.2.3 Camera Projection Model 66
• 5.2.4 LiDAR Measurement Model 67
• 5.2.5 GNSS/RTK Observation Model 67
• 5.3 Online Real-Time Localization Layer. 68
• 5.4 Semantic Optimization Layer 70
• 5.4.1 Semantic Constraint for Global Optimization 71
• 5.4.2 Multilayer Object Graph Matching 74
• 5.4.3 Semantic global optimization 76
• 5.4.4 Expansion to Cooperative Perception 78
• Chapter 6. Experiments 81
• 6.1 Experimental Setup 81
• 6.2 Localization Accuracy and Robustness 86
• 6.2.1 Evaluation under Environmental Variations and Sensor Failures 87
• 6.2.2 Comparison with Other Localization Methods 91
• 6.3 Cooperative Perception Extension Experiment 92
• 6.4 Validation of SMF Effectiveness 93
• 6.4.1 Semantic Environmental Modeling for Experiments 94
• 6.4.2 Impact of Semantic Rules on Performance 94
• 6.4.3 Ablation Study on the Influence of Semantic Factors 97
• 6.5 Hierarchical Scene Representation 99
• 6.5.1 Datasets and Implementation Details 99
• 6.5.2 Evaluation of Low-Level Features for SLAM and Localization 100
• 6.5.3 Object-Level Semantic Perception Evaluation 104
• 6.5.4 Scene-Level Semantic Perception Evaluation 106
• 6.5.5 Evaluation of Runtime Efficiency in Localization 109
• 6.6 Discussion 111
• Chapter 7. Conclusion and Future Works 113
• 7.1 Conclusion 113
• 7.2 Future Works 114
• References 116