OpenMonoGS-SLAM : monocular gaussian splatting SLAM with open-set semantics

다국어 초록 (Multilingual Abstract)

Simultaneous Localization and Mapping (SLAM) is a foundational component in robotics, AR/VR, and autonomous systems. With the rising focus on spatial AI in recent years, combining SLAM with semantic understanding has become increasingly important for enabling intelligent perception and interaction. Recent efforts have explored this integration, but they often rely on depth sensors or closed-set semantic models, limiting their scalability and adaptability in open-world environments. In this work, we present OpenMonoGS-SLAM, the first monocular SLAM framework that unifies 3D Gaussian Splatting (3DGS) with open-set semantic understanding. To achieve our goal, we leverage recent advances in Visual Foundation Models (VFMs), including MASt3R for visual geometry and SAM and CLIP for open-vocabulary semantics. These models provide robust generalization across diverse tasks, enabling accurate monocular camera tracking and mapping, as well as a rich understanding of semantics in open-world environments. Our method operates without any depth input or 3D semantic ground truth, relying solely on self-supervised learning objectives. Furthermore, we propose a memory mechanism specifically designed to manage high-dimensional semantic features, which effectively constructs Gaussian semantic feature maps, leading to strong overall performance. Experimental results demonstrate that our approach achieves performance comparable to or surpassing existing baselines in both closed-set and open-set segmentation tasks, all without relying on supplementary sensors such as depth maps or semantic annotations.

국문 초록 (Abstract)

Simultaneous Localization and Mapping(SLAM)은 로보틱스, AR VR, 자율주행 시스템의 핵심 기술이며, 최근 Spatial AI의 부상과 함께 의미론적 이해를 결합한 Semantic SLAM의 중요성이 커지고 있다. 그러나 기존 방법들은 깊이 센서나 폐쇄 집합(closed-set) 의미론 모델에 의존하는 경우가 많아, 오픈월드 환경에서의 확장성과 적응성에 한계가 있다. 본 연구에서는 3D Gaussian Splatting과 오픈셋 의미론 이해를 통합한 최초의 단안 SLAM 프레임워크 OpenMonoGS-SLAM을 제안한다. MASt3R로 기하 정보를 추정하고, SAM과 CLIP으로 언어적 의미 정보를 학습하여, 깊이 입력이나 3D 의미 정보 없이도 자기지도 학습만으로 추적·매핑과 의미론적 표현을 동시에 수행한다. 또한 고차원 의미론적 특징을 효율적으로 관리하는 메모리 메커니즘을 도입해 Gaussian 의미론 특징 맵을 안정적으로 구성한다. 실험 결과, 제안 방법은 추가 센서나 주석 없이도 폐쇄 집합 및 오픈셋 분할 과제에서 기존 기준 방법들과 대등하거나 더 우수한 성능을 달성한다.

목차 (Table of Contents)

• Chapter 1. Introduction 1
• Chapter 2. Related Work 5
• 1. Visual Foundation Model 5
• 2. Visual SLAM 6
• Chapter 3. Mehtod 8
• A. 3DGS-based Monocular SLAM 9
• 1) MASt3R-SLAM 9
• 2) 3D Gaussian Representation 10
• B. Multi-Scale Semantic Learning 11
• C. Language-embedded Semantic Memory Bank 12
• D. Self-Supervised Learning Objectives 13
• a) Photometric Supervision 14
• b) Multi-View Semantic Consistency 14
• c) Language-Guided Semantic Alignment 14
• d) Final Loss 15
• Chapter 4. Experiments 16
• 1. Experiments Setup 16
• 1) Implementation Details 16
• 2) Dataset and Evaluation Metrics 17
• 3) Baselines 17
• 2. Results 18
• 1) Novel View Synthesis 18
• 2) Camera Tracking 20
• 3) Open-set Segmentation Results 21
• 4) Closed-set Segmentation Results 23
• 5) Ablations and Analysis 25
• Chapter 5. Conclusion 27
• References 28
• Appendix 37
• 1. Architecture 37
• 2. Hyperparameter setting 38
• 3. Runtime Evaluation 39
• 4. Class-agnostic prompt segmentation 40
• Korean Abstract 42