Non-rigid Mesh-to-RGBD Registration Using Visual-Geometric Foundation Models and Neural Descriptor Field = 시각-기하 파운데이션 피쳐 기반 뉴럴 디스크립터 필드를 활용한 강건한 비강체 Mesh-to-RGBD 정합 연구

다국어 초록 (Multilingual Abstract)

Non-rigid registration is crucial for robot manipulation and 3D vision, yet ensuring stable performance with a single RGB-D sensor is challengingdue to partial observations and occlusions. Conventional optimization-based methods typically rely solely on geometric features, often leading to registration failures. To address this, we propose a novel framework based on Neural Descriptor Field (NDF) that leverages pre-trained visual-geometric foundation models to construct a continuous 3D descriptor field via instance-specific optimization. Our registration process consists of two stages: a deformation-tolerant coarse pose estimation based on feature similarity, followed by test-time optimization that maximizes feature consistency and geometric alignment. Validated on a new synthetic Mesh-to-Scene dataset, our method achieves superior accuracy and robustness compared to state-of-the-art techniques, even under significant pose variations and shape deformations.

국문 초록 (Abstract)

비강체 정합(Non-rigid Registration)은 로봇 조작 및 3D 비전 분야의 핵심 기술이나, 단일 RGB-D 센서를 이용한 환경에서는 부분적인 관측과 가림(Occlusion) 현상으로 인해 안정적인 성능 확보가 어렵다. 기존의 최적화 기반 방법들은 주로 기하학적 특징에만 의존하여 특징이 부족한 영역에서 정합 실패가 빈번하게 발생한다는 한계가 있다. 본 논문에서는 이러한 문제를 해결하기 위해 사전 학습된 시각-기하학적 파운데이션 모델(Visual-geometric foundation models)을 활용하여 인스턴스별 연속적 3D 디스크립터 필드를 구축하는 뉴럴 디스크립터 필드(Neural Descriptor Field, NDF) 기반의 새로운 프레임워크를 제안한다. 제안하는 정합 프로세스는 특징 유사도에 기반한 변형에 강건한 초기 포즈 추정(Coarse pose estimation)과, 특징 일관성 및 기하학적 정렬을 극대화하는 테스트 시점 최적화(Test-time optimization)의 두 단계로 구성된다. 자체 구축한 Mesh-to-Scene 합성 데이터셋을 통한 실험 결과, 제안 기법은 큰 포즈 변화와 형상 변형이 존재하는 조건에서도 기존
최적화 기반 SOTA(State-of-the-art) 기법들보다 우수한 정확도와 강건성을 입증하였다.

목차 (Table of Contents)

• Abstract i
• Table of Contents ii
• List of Tables iv
• List of Figures v
• Chapter 1. Introduction 1
• 1.1. Motivation 1
• 1.2. Problem Statement 2
• 1.3. Research Objectives and Contribution 3
• Chapter 2. Related Work 4
• 2.1. Geometry-based Non-rigid Registration 4
• 2.2. Neural Implicit Representation for Deformable Objects 4
• 2.3. CAD-based Object Pose Estimation 5
• 2.4. Visual and Geometric Foundation Models in 3D Vision 6
• Chapter 3. Background and Preliminaries 8
• 3.1. Visual Foundation Model: DINOv2 and Featup 8
• 3.2. Geometric Foundation Model: GeDi 8
• 3.3. Neural Implicit Fields and Hash-Grid Encoding 9
• Chapter 4. Method 10
• 4.1. System Overview 10
• 4.2. Onboarding Stage 10
• 4.2.1. Multi-view Feature Extraction and Integration 10
• 4.2.2. Network Architecture and Loss Functions for NDF 12
• 4.3. Coarse Pose Estimation via Feature Similarity Scoring 13
• 4.3.1. Feature Correspondence Generation 14
• 4.3.2. NDF-based Pose Scoring 14
• 4.4. Non-rigid Registration 15
• 4.4.1. Deformation Field Modeling Using MLP 15
• 4.4.2. NDF Feature Similarity 15
• 4.4.3. Feature Consistency-based Soft Chamfer Distance 16
• 4.4.4. Correspondences 17
• Chapter 5. Mesh-to-Scene Synthetic Dataset Generation 18
• 5.1. Object Selection and Physical Deformation Simulation 18
• 5.1.1. Motivation for Synthetic Dataset 18
• 5.1.2. Object Selection 18
• 5.1.3. Deformation Method 18
• 5.1.4. Ground-truth Correspondence Generation 19
• 5.2. Realistic Scene Rendering and Occlusion Modeling 20
• Chapter 6. Experiments and Results 23
• 6.1. Experimental Setup and Evaluation Metrics 23
• 6.1.1. NDF Onboarding 23
• 6.1.2. Deformation Field Optimization 23
• 6.2. Evaluation Metrics 23
• 6.3. Comparative Analysis of Coarse Alignment Performance 24
• 6.4. Quantitative Evaluation of Non-rigid Registration 25
• 6.5. Robustness to Severe Occlusions 27
• 6.6. Qualitative Results and Semantic Consistency 27
• Chapter 7. Discussion 30
• 7.1. Ablation Study on Loss Components 30
• 7.2. Computation Efficiency and Memory Trade-offs 31
• 7.3. Limitations and Failure Case Analysis 32
• Chapter 8. Conclusion 33
• 8.1. Summary of Research 33
• 8.2. Future Direction 33
• Bibliography 35
• 국문 초록 38