Adaptive Spatial Token Reduction for Efficient Video Vision Transformers

다국어 초록 (Multilingual Abstract)

The objective of this study was to explore a training-free method to reduce tokens and consequently computational costs in vision transformers in a way that is spatially coherent. Vision transformers （ViTs） use self-attention to achieve state-of-the-art results in image and video intelligence tasks, including classification, action recognition, text-to-video retrieval （and vice versa）, etc. The input to a ViT is always converted into a sequence of ‘tokens’ processed within the layers of a transformer. The self-attention inside a transformer is of quadratic complexity with respect to token count. The higher the number of tokens, the better the results of the transformer, but the higher the computational cost. Methods to reduce token count by pruning or merging have been introduced. These methods perform token merging or pruning based on attention matrix data and are performed anywhere in the sequence. This thesis investigates a spatially-constrained token reduction strategy, inspired the concept that spatially close together tokens carry similar information, especially if they are background or low-motion tokens, aiming to study if if reducing tokens while being mindful of their original location in the image or frame can contribute to improving efficiency without sacrificing accuracy. This work proposes a spatially- constrained mechanism that is injected in between transformer layers once, and replaces the transformer’s view with an adaptive view where tokens are different sizes based on the data they contain. This coarse+fine view aims to represent important areas at the original resolution, but convert less important regions into singular tokens at 2x2 coverage. The selected importance metric is L2 Norm of token embeddings, and the original position of each token is tracked throughout the network. The proposed method achieves only a 0.09% accuracy drop at 128 GFLOPs on Kinetics-400, and 0.24% accuracy drop at 115 GFLOPs on Something-Something V2.
Key words
Video Transformers, Token Merging, Token Reduction, Efficient Inference, Spatial Token Merging, VideoMAE

목차 (Table of Contents)

• 1 Introduction 1
• 1.1 Motivation 1
• 1.2 Problem Statement 2
• 1.3 Contributions 3
• 2 Related Work 4
• 2.1 Token Pruning 5
• 2.2 Token Merging 6
• 2.3 Token Clustering and Adaptive Spatial Tokenization 7
• 3 Methodology 7
• 3.1 Method Overview 8
• 3.2 Importance Metric Selection 9
• 3.3 Importance signals for training-free reduction 11
• 3.4 Coarse/Fine Classification 11
• 3.5 Position Mask 12
• 3.6 Dynamic Spatio-temporal Adaptation 13
• 4 Evaluation 13
• 4.1 Experimental Setup 13
• 4.2 Main Results on VideoMAE-B 15
• 4.3 Generalization and Ablation Studies 20
• 4.4 Qualitative Results 28
• 5 Discussion 30
• 5.1 Limitations 30
• 5.2 Future Work 31
• 5.2.1 Extension to Multimodal Vision-Language Models 31
• 6 Conclusion 32
• 7 References 33