어텐션 기반 피처 융합 모듈과 그래드-캠의 설명 가능성을 적용한 비디오 보조 심판 시스템 연구 = Attention-Based Feature Fusion Modules and Grad-CAM Explainability for an Automated VAR System

다국어 초록 (Multilingual Abstract)

The Video Assistant Referee (VAR) system has played a crucial role in enhancing the fairness and accuracy of soccer officiating. However, its continued reliance on human intervention, inconsistent decision-making, and high operational costs reveal inherent limitations that motivate the development of a fully automated VAR system. A fundamental technical challenge in such a system lies in effectively aggregating information from synchronized multi-view video streams while maintaining both high recognition performance and decision transparency.

This thesis investigates the optimal design of attention-based feature aggregation for automated multi-view foul recognition. We propose and systematically analyze three distinct attention-based fusion architectures—Attention Fusion Module (AFM), Pooling-Guided Fusion Module (PGFM), and Hierarchical Attention Fusion Module (HAFM)—built upon an MViT-v2-S backbone to extract spatio-temporal representations from each camera view. Extensive experiments conducted on the SoccerNet-MVFouls dataset demonstrate that the Attention-First strategy, which explicitly models inter-view interactions before feature summarization, achieves the best performance. In particular, AFM attains a Test Leaderboard Value of 41.92, outperforming the baseline model by 11.98\% and consistently surpassing the other proposed fusion strategies.

Beyond quantitative improvements, this work emphasizes explainability as a core requirement for automated officiating systems. By integrating Layer-Grad-CAM into the proposed framework, we provide a qualitative analysis of the model’s decision-making process. Visualization results reveal that when correct predictions are made with high confidence, the model focuses on critical spatio-temporal regions such as player contact points and ball interactions, closely resembling human referee behavior. Conversely, high-confidence misclassifications expose a tendency toward over-generalization, where attention diffuses across global scene context rather than specific foul-related actions.

Overall, this study demonstrates that attention-first multi-view aggregation not only maximizes recognition accuracy but also enables meaningful visual explanations of automated decisions. These findings establish a practical foundation for the development of reliable and explainable automated VAR systems and offer insights into future research directions aimed at mitigating model bias and improving robustness in complex, real-world sports scenarios.

목차 (Table of Contents)

• 1. 서론 1
• 2. 관련 연구 4
• 2.1. 비디오 기반 행동 인식 (Video-based Action Recognition) 4
• 2.2. 다중 시점 행동 인식과 어텐션 기반 정보 융합 (Multi-view Action Recognition and Attention-based Fusion) 6
• 2.3. 어텐션 메커니즘과 트랜스포머 기반 시각 모델 7
• 2.4. 설명 가능한 인공지능 (XAI) in Computer Vision 7
• 3. 제안하는 방법 (Proposed Method) 9
• 3.1. 전체 시스템 구조 (Overall System Architecture) 9
• 3.2. 어텐션 기반 피처 퓨전 모듈 (Attention-Based Feature Fusion Modules) 12
• 3.2.1. AFM (Attention Fusion Module) 12
• 3.2.2. PGFM (Pooling-Guided Fusion Module) 14
• 3.2.3. HAFM (Hierarchical Attention Fusion Module) 15
• 3.3. Grad-CAM 기반 설명 가능성 확보 17
• 3.3.1. CAM과 Grad-CAM의 구조적 차이 및 범용성 18
• 3.3.2. Grad-CAM의 활성화 맵 생성 과정 18
• 3.3.3. Layer-Grad-CAM 적용 전략 20
• 4. 실험 (Experiments) 22
• 4.1. 실험 환경 (Experimental Setup) 22
• 4.1.1. 데이터셋 (Dataset) 22
• 4.1.2. 기반 모델 23
• 4.1.3. 평가지표 24
• 4.1.4. 베이스라인 24
• 4.2. 어텐션 기반 피처 융합 모듈의 성능 평가 결과 25
• 4.3. 정량적 성능 평가에 대한 분석 및 소결 26
• 4.3.1. 어텐션 기반 피처 융합 모듈별 성능 분석 26
• 4.3.2. 태스크 특성에 따른 성능 차이 분석 27
• 4.4. Grad-CAM 기반 설명 가능성의 정성적 분석 28
• 4.4.1. 분석 방법 29
• 4.4.2. 히트맵 (Heatmap) 생성 방법 31
• 4.4.3. 시각화 결과 및 분석 32
• 5. 결론 (Conclusion) 35
• 참고문헌 38
• ABSTRACT 42
• 부록 44
• 부록 (가) 정답을 예측한 Grad-CAM 시각화 히트맵 44
• 부록 (나) 샘플 27의 원거리 시점 멀티뷰 Grad-CAM 시각화 히트맵 44
• 부록 (다) SoccerNet 2024 Multi-View Foul Recognition challenge leaderboard 결과 45