Multi-View Perception: 2D Object Detection and Zero-Shot Instance Association = 다중 뷰 환경에서의 2D 객체 검출 및 제로샷 인스턴스 매칭

국문 초록 (Abstract)

Multi-view perception (다중 뷰 인식)은 여러 카메라에서 캡처한 이미지를 활용하여 복잡한 장면을 이해하는 것을 목표로 한다. Multi-view 2D object detection (다중 뷰 2D 객체 검출)과 multi-view instance association (다중 뷰 인스턴스 매칭)은 multi-view perception에 속하는 대표적인 두 가지 연구 분야이다. 먼저, multi-view 2D object detection은 심각한 occlusion이 발생하는 혼잡한 환경에서 여러 시점에서 찍힌 객체의 클래스와 2D 상자를 찾아내는 것을 목표로 한다. 다음으로, multi-view instance association은 카메라의 시점 차이로 인해 같은 object의 외관이 서로 다를 때, 서로 다른 카메라 뷰에서 동일한 객체를 매칭시킨다. Multi-view perception은 소매점에서의 자동 결제, 다중 카메라 추적, 로봇 작동과 같은 실제 응용 분야에서 중요하게 활용된다.
최근 multi-view perception model들이 많이 제안되고 있지만, 여전한 한계들이 존재한다. 먼저, query-based multi-view object detector는 모든 view에서 하나의 쿼리를 공유해서 사용한다. 이런 방식은 각 뷰의 시각적, 기하학적 특성을 하나로 압축시켜버려서 여러 뷰의 고유성이 사라지는 문제가 있다. 또한, query-based 방식에서 사용되는 hierarchy-based multi-view feature fusion 방식은 같은 3D point에서 추출된 특징들만을 fusion하여 모든 뷰가 그 정보를 복제하여 사용한다. 그로 인해 각 뷰의 고유성이 사라지고, 주요 특징에 포함되지 않는 3D point의 주변 특징 정보는 무시된다는 한계가 있다. 이뿐만 아니라, multi-view association model들을 트레이닝시키기 위해서는 대량의 라벨이 필요하다. 이 문제를 해결하기 위해 multimodal large language model (멀티모달 대형 언어 모델) 기반의 zero-shot instance association (제로샷 인스턴스 매칭) 방법을 이용하여 이미지 간 시각 정보를 비교하고, epipolar constraint를 사용해서 geometric 정보를 사용할 수 있다. 하지만, 이 방식은 epipolar line 주변에 여전히 많은 후보군이 존재한다는 점과 viewing angle difference에 따라 object의 시각적 특징이 많이 달라진 점을 간과한다.
이런 문제들을 해결하기 위해, 이 논문은 세 가지 해결책을 제안한다. 첫째로, 공유 쿼리 문제를 해결하기 위해 각 뷰의 전용 쿼리를 제안하고, geometrically binding된 여러 뷰의 query를 fusion하는 방식을 제안한다. 이런 방식은 각 뷰의 고유성을 보존하면서도 view 간 fusion을 가능하게 한다. 이를 통해 shared query 방식 및 multi-view 2D object detection baseline 대비 큰 폭의 성능 향상을 거두었다. 둘째로, primary-feature-only fusion의 문제를 해결하기 위해 camera position 및 viewing direction vector embedding을 통한 multi-view subordinate feature fusion 방식을 제안한다. 이런 방식은 각 뷰의 viewpoint 특성을 이해하면서 primary와 subordinate feature를 fusion하는 것을 가능하게 한다. 이를 통해 primary feature만을 fusion할 때보다 높은 성능을 성취하였다. 셋째로, supervised multi-view instance association의 labeling 문제를 해결하기 위해 트레이닝이 필요 없는 zero-shot multi-view instance association framework을 제안한다. 제안 방식은 멀티모달 대형 언어 모델을 통해 물체의 position 정보를 추출하고, 두 카메라 간 viewing angle 차이에 따라 물체의 visual 정보와 position 정보를 adaptively fusion한다. 이를 통해 label을 사용하지 않고도 일부 supervised 모델의 성능을 제쳤고, zero-shot baseline 중 가장 높은 성능을 거두었다.

다국어 초록 (Multilingual Abstract)

Multi-view perception aims to understand complex scenes by utilizing images captured from multiple cameras. Multi-view 2D object detection and multi-view instance association are two representative research topics in multi-view perception. First, multi-view 2D object detection aims to find the classes and 2D boxes of objects across multiple cameras in crowded environments where severe occlusion occurs. Next, multi-view instance association matches the same objects in different camera views where the appearance of the object differs due to variations in camera viewpoint. Multi-view perception is important in real-world applications such as automated retail checkout, multi-camera tracking, and robotic operation.
Recently, many multi-view perception models have been proposed, but there are still limitations. First, the query-based multi-view object detector shares a single query across all views. This method compresses the visual and geometric characteristics of each view into one representation, and therefore the uniqueness of each view disappears. Second, the hierarchical multi-view feature fusion method used in query-based approaches fuses only features extracted from the same 3D point, and all views replicate the shared feature. As a result, the surrounding features of the primary 3D point are ignored. Third, a large amount of labels is required to train multi-view association methods. To solve this problem, the zero-shot instance association method based on a multimodal large language model (MLLM) can be used to compare the visual features between images, and geometric constraints such as epipolar geometry can be applied. However, there are still many matching candidates near the epipolar line, and the visual features of the object differ greatly depending on the viewing-angle differences.
To solve these problems, this paper proposes three solutions. First, to address the shared-query problem, this paper introduces a view-specific query for each view with a strategy to fuse the geometrically bound queries of multiple views. This method makes it possible to perform cross-view fusion while preserving the uniqueness of each view. This results in a significant performance improvement over the shared-query method and the multi-view 2D object detection baseline. Second, to solve the problem of primary-feature-only fusion, we propose a multi-view subordinate feature fusion method using camera position and viewing vector embeddings. This method fuses primary and subordinate features together while capturing the viewpoint characteristics of each view. This achieves higher performance compared to the fusion of only primary features. Third, to address the labeling issue of supervised multi-view instance association, a zero-shot multi-view instance association framework is proposed. The proposed method extracts the object's position information through an MLLM and adaptively fuses the object's visual and positional information according to the viewing-angle difference between the two cameras. This outperforms some supervised models without the use of labels and achieves the highest performance among zero-shot baselines.

목차 (Table of Contents)

• Abstract i
• Contents iii
• List of Figures vii
• List of Tables ix
• 1 Introduction 1
• 1.1 Background 1
• 1.2 Overview 4
• 1.3 Thesis outline 6
• 2 MVQT: Multi-View Query Transformer for 2D Object Detection 7
• 2.1 Introduction 7
• 2.2 Previous works 11
• 2.2.1 Perspective projection–based methods 11
• 2.2.2 Pairwise ReID–based methods 11
• 2.2.3 Query–based methods 12
• 2.3 Method 13
• 2.3.1 Query-based multi-view 2D object detection baseline 13
• 2.3.1.1 Query 15
• 2.3.1.2 Multi-view image features 15
• 2.3.1.3 Output 16
• 2.3.1.4 Loss 16
• 2.3.1.5 Meaning of attention weights 17
• 2.3.2 View-specific query 17
• 2.3.2.1 Issue of query-based framework 17
• 2.3.2.2 View-specific query framework 19
• 2.3.3 Geometry-aligned feature fusion 23
• 2.3.3.1 Issue of view-specific query framework 23
• 2.3.3.2 Geometry-aligned feature fusion framework 25
• 2.4 Experiments 27
• 2.4.1 Experimental configurations 27
• 2.4.1.1 Dataset 27
• 2.4.1.2 Metric 28
• 2.4.1.3 Implementation details 28
• 2.4.2 Comparison with pairwise re-identification-based multi-view 2D object detection models 28
• 2.4.3 Comparison with perspective projection-based multi-view 2D object detection models 32
• 2.4.4 Comparison with query-based multi-view 2D object detection models 36
• 2.4.5 Why MVQT improves MODA more than F1 37
• 2.4.6 Effect of calibration augmentation on robustness to calibration noise 40
• 2.4.7 Computational and memory complexity 46
• 2.4.8 Scalability 48
• 2.5 Conclusion 48
• 3 SFF3D: Multi-View Subordinate Feature Fusion with 3D Position Embedding 51
• 3.1 Introduction 51
• 3.2 Previous works 54
• 3.2.1 Sequential fusion 55
• 3.2.2 Joint fusion 56
• 3.3 Method 57
• 3.3.1 Multi-view feature fusion baseline in MVQT 57
• 3.3.2 Multi-view subordinate feature fusion with 3D positional embedding (SFF3D) 59
• 3.3.2.1 3D positional embedding for geometry-aware fusion 62
• 3.3.2.2 Camera viewing direction embedding 63
• 3.4 Experiments 64
• 3.4.1 Experimental configurations 64
• 3.4.1.1 Dataset 64
• 3.4.1.2 Metric 64
• 3.4.1.3 Implementation details 65
• 3.4.2 Multi-view feature fusion baselines 65
• 3.4.2.1 MVCNN 65
• 3.4.2.2 GVCNN 65
• 3.4.2.3 CMVD 66
• 3.4.3 Performance comparison with multi-view feature fusion baselines 66
• 3.4.4 Ablation study on positional embedding 68
• 3.4.5 Ablation study on camera viewing direction embedding 72
• 3.4.6 Analysis of attention distribution across views 73
• 3.5 Conclusion 74
• 4 ZS-MVIA: Zero-Shot Multi-View Instance Association 76
• 4.1 Introduction 76
• 4.2 Previous works 78
• 4.2.1 Supervised multi-view instance association 78
• 4.2.2 Language model-based visual comparison 80
• 4.2.2.1 Pretrained vision and language models 80
• 4.2.2.2 Language model-based visual comparison 81
• 4.2.3 Positional cue extraction from images 81
• 4.3 Method 82
• 4.3.1 ZS-MVIA baseline 84
• 4.3.1.1 Visual comparison 84
• 4.3.1.2 Positional comparison 84
• 4.3.1.3 Distance combining visual and positional comparisons 85
• 4.3.2 RPI-based 3D position recovery 85
• 4.3.2.1 Challenges in estimating 3D object position using MLLM 85
• 4.3.2.2 3D point recovery via ray-plane intersection (RPI) 88
• 4.3.2.3 Visual and positional distance fusion via reciprocal rank fusion (RRF) 89
• 4.3.3 Angle-adaptive reciprocal rank fusion (AA-RRF) 90
• 4.4 Experiments 93
• 4.4.1 Experimental configurations 93
• 4.4.1.1 Dataset 93
• 4.4.1.2 Metric 94
• 4.4.1.3 Implementation details 94
• 4.4.2 Performance comparison with ZS-MVIA baselines 94
• 4.4.3 Heatmap of instance association distance 97
• 4.4.4 Ablation study on angle-adaptive weight in AA-RRF 99
• 4.4.5 Ablation study on fusion weights in RRF 100
• 4.5 Conclusion 101
• 5 Conclusion 102
• Bibliography 104
• 초 록 113