Enhancing 3D Visual Grounding in VLMs via Multi-Modal Feature Fusion and Numerical Proximity-Aware Loss Function = 대규모 언어-시각 모델의 3D 시각적 그라운딩 성능 향상을 위한 멀티모달 특징 융합 및 수치적 근접성 기반 손실 함수

다국어 초록 (Multilingual Abstract)

While recent rapid advancements in large-scale Vision-Language Models (VLMs) have significantly contributed to AI's integrated understanding of visual information and natural language, distinct limitations remain in the field of 3D Visual Grounding (3DVG), which extends this capability to 3D physical space. Existing VLM-based approaches suffer from geometric ambiguity, losing range and spatial structure information during the process of projecting 3D scenes into 2D images. Furthermore, they face issues with degraded numerical precision due to classification-based optimization methods that treat continuous physical coordinates as independent text tokens.
To overcome these structural limitations, this study proposes an integrated framework that effectively injects 3D spatial awareness while preserving the powerful semantic reasoning capabilities of 2D VLMs. The core methodology of this paper can be summarized into two main aspects.
First, unlike existing simple concatenation methods, we introduce a Structure-Preserving Multi-modal Fusion strategy. We extract geometric features and dense semantic features, then hierarchically inject them into the VLM's visual encoder via a Zero-Initialized Adapter. Additionally, we introduce Geometric QA as an auxiliary task to force the model to encode the injected geometric features as key reasoning cues rather than simple noise. This technique induces the model to organically integrate and learn 2D visual information and 3D geometric information without compromising the parameter distribution of the pre-trained VLM, thereby preventing catastrophic forgetting.
Second, to address the numerical discontinuity in coordinate prediction, we introduce the Place-value weighted Earth Mover's Distance (EMD) Loss. This serves as a Numerical Proximity-Aware Loss that directly reflects the physical distance between the prediction distribution and the ground truth in the cost function. It helps the model internalize the ordinal relationship and sense of physical distance between digit tokens, aiding convergence to precise locations.
Extensive experiments were conducted on major benchmark datasets such as ScanRefer, Nr3D, and Sr3D to verify the effectiveness of the proposed model. Experimental results show that the proposed method achieved an accuracy of 74.6% on the Nr3D dataset, surpassing the previous state-of-the-art by 4.7 percentage points, and recorded 61.0% on ScanRefer, demonstrating its superiority. In particular, it demonstrated exceptional robustness and generalization performance even in complex environments where distinguishing between objects is difficult and in unseen scenes not used for training. By presenting geometric scaffolding and optimization techniques to evolve 2D VLMs into 3D spatial reasoning experts, this study suggests a new direction for visual grounding research for Embodied AI.

국문 초록 (Abstract)

최근 대규모 Vision-Language Models (VLMs)의 비약적인 발전은 인공지능이 시각 정보와 자연어를 통합적으로 이해하는 데 크게 기여하였으나, 이를 3차원 물리 공간으로 확장하는 3D Visual Grounding (3DVG) 분야에서는 여전히 뚜렷한 한계가 존재한다. 기존의 VLM 기반 접근법들은 3D 장면을 2D 이미지로 투영하여 처리하는 과정에서 거리(Range)와 공간 구조(Spatial Structure) 정보를 소실하는 기하학적 모호성(Geometric Ambiguity)을 겪으며, 연속적인 물리 좌표를 독립적인 텍스트 토큰으로 취급하는 분류 기반 최적화 방식으로 인해 수치적 정밀도(Numerical Precision) 가 저하되는 문제를 안고 있다.
본 연구는 이러한 구조적 한계를 극복하기 위해, 2D VLM의 강력한 의미론적 추론 능력을 보존하면서 3차원 공간 인지 능력을 효과적으로 주입하는 통합 프레임워크를 제안한다. 본 논문의 핵심 방법론은 두 가지로 요약된다.
첫째, 기존의 단순 결합 방식과 달리 구조 보존형 멀티모달 융합(Structure-Preserving Multi-modal Fusion) 전략을 도입하였다. 본 논문에서는 기하학적 특징(Geometric Features)과 전역적 의미 정보(Dense Semantic Features)를 추출하고, 이를 Zero-Initialized Adapter를 통해 VLM의 시각 인코더에 계층적으로 주입한다. 또한, 모델이 주입된 기하학적 특징을 단순 노이즈가 아닌 핵심 추론 단서로 인코딩하도록 강제하기 위해 Geometric QA를 보조 과제로 도입하였다. 이 기법은 사전 학습된 VLM의 파라미터 분포를 훼손하지 않으면서(Catastrophic Forgetting 방지), 모델이 2D 시각 정보와 3D 기하 정보를 유기적으로 통합하여 학습하도록 유도한다.
둘째, 좌표 예측의 수치적 불연속성을 해결하기 위해 Place-value weighted Earth Mover's Distance (EMD) Loss를 도입하였다. 이는 예측 분포와 정답 간의 물리적 거리를 비용 함수에 직접 반영하는 수치적 인접성 기반 손실(Numerical Proximity-Aware Loss)로써, 모델이 숫자 토큰 간의 서수적 관계(Ordinality)와 물리적 거리감을 내재화하고 정밀한 위치로 수렴하도록 돕는다.
제안된 모델의 유효성을 검증하기 위해 ScanRefer, Nr3D, Sr3D 등 주요 벤치마크 데이터셋에서 광범위한 실험을 수행하였다. 실험 결과, 제안하는 방법은 Nr3D 데이터셋에서 74.6%의 정확도를 달성하며 기존 최고 성능(State-of-the-Art)을 4.7%p 경신하였고, ScanRefer에서도 61.0%를 기록하며 우수성을 입증하였다. 특히 객체 간 구분이 어려운 복잡한 환경과 학습에 사용되지 않은 새로운 장면(Unseen Scenes)에서도 탁월한 강건성(Robustness)과 일반화(Generalization) 성능을 입증하였다. 본 연구는 2D VLM을 3D 공간 추론 전문가로 진화시키기 위한 기하학적 스캐폴딩과 최적화 기법을 제시함으로써, Embodied AI를 위한 시각적 그라운딩 연구의 새로운 방향성을 제시한다.

목차 (Table of Contents)

• Abstract i
• Chapter 1 Introduction 1
• 1.1 Background 1
• 1.2 Problem Statement 3
• 1.3 Proposed Approach 4
• 1.4 Key Contributions 5
• Chapter 2 Related Works 7
• 2.1 3D Visual Grounding (3DVG) 7
• 2.1.1 Traditional Supervised Approaches: From Point Clouds to Language 7
• 2.1.2 VLM-based Approaches: Leveraging 2D Priors for 3D 8
• 2.1.3 Limitations in Multi-modal Fusion and Supervision 9
• 2.2 Multi-modal Learning in 3D 9
• 2.2.1 Multi-modal Learning in 3D: From Concatenation to Adaptive Fusion 9
• 2.2.2 Structural Conditioning via Zero-Initialization 10
• 2.3 Parameter-Efficient Fine-Tuning (PEFT) for 3D Adaptation 11
• 2.3.1 LoRA and Hybrid Adaptation 11
• 2.4 Numerical Reasoning and Regression in Large Language Models 11
• 2.4.1 The Paradox of “Text Generation as Regression” 12
• Chapter 3 Method 14
• 3.1 Overview 14
• 3.2 Preliminary: RGB-based Panoramic Grounding Framework 17
• 3.2.1 The Framework: Render, Ground, and Lift 17
• 3.2.2 Problem Statement: The Gap between 2D Perception and 3D Reality 18
• 3.3 Multi-modal Panoramic Representations 19
• 3.3.1 Geometric Feature Map Construction 20
• 3.3.2 Semantic Feature Map Construction via 3D Lifting 20
• 3.4 Multi-Modal Feature Fusion via Zero Initialized Adapter 22
• 3.4.1 Adapter Architecture with Zero-Initialization 22
• 3.4.2 Hierarchical Injection Strategy 25
• 3.5 Training Objectives and Multi-task Learning Strategy 26
• 3.5.1 Numerical Proximity-Aware Loss Function via Place-value Weighted EMD 27
• 3.5.2 Auxiliary Task: Geometric QA for Explicit Geometric Feature Encoding 29
• Chapter 4 Experiments 32
• 4.1 Experimental Setup 32
• 4.1.1 Datasets 32
• 4.1.2 Evaluation Metrics 33
• 4.1.3 Implementation Details 34
• 4.2 Comparison with State-of-the-Art Methods 35
• 4.2.1 Performance Analysis on ReferIt3D (Nr3D & Sr3D) 35
• 4.2.2 Performance Analysis on ScanRefer 37
• 4.3 Ablation Study: Impact of Proposed Components 38
• 4.3.1 Effectiveness of Multi-Modal Feature Fusion 40
• 4.3.2 Effectiveness of Numerical Proximity-Aware Loss 41
• 4.3.3 Impact of Adapter Design Strategy 41
• 4.4 Qualitative Analysis 42
• 4.4.1 Impact of Adapter Design Strategy 42
• 4.4.2 Successful Grounding in Complex Scenarios 44
• 4.4.3 Comparison with Baseline Methods 46
• 4.5 Robustness and Generalization 46
• 4.5.1 Generalization to Unseen 3D Scenes 46
• Chapter 5 Conclusion 51
• 5.1 Summary of Research 51
• 5.2 Key Empirical Findings 53
• 5.2.1 Critical Role of Multi-modal Feature Fusion 53
• 5.2.2 Precision Improvement via Numerical Proximity-Aware Loss 54
• 5.2.3 Robustness and Generalization 54
• 5.3 Limitations 55
• 5.4 Future Work 56
• Abstract in Korean 64