DICE : disentangled instance-class knowlEdge prompt tuning via SAE for vision-language models

다국어 초록 (Multilingual Abstract)

CLIP, known for its strong zero-shot generalization, has received significant attention in prompt tuning for its ability to adapt effectively to new tasks in few-shot settings while mitigating overfitting. Recent studies have employed descriptions generated by Large Language Models (LLMs) as predefined prompts to provide richer class-level semantics. However, because LLMs are not directly grounded in visual information, predefined descriptions often misalign with an image’s visual semantics, yielding generic class-level phrases instead of instance-specific descriptions. As a result, fine-grained cues essential for discrimination are overlooked, limiting generalization in few-shot settings where both class-level and instance-level knowledge are crucial. To this end, we propose Disentangled Instance-Class knowlEdge (DICE) prompt tuning, a novel framework that treats both class-level and instance-level knowledge from LLMs. DICE leverages a Sparse Autoencoder (SAE) to disentangle LLM-derived knowledge into instance-level components, which provides instance-specific concept vectors. These concept vectors are then fused with class embeddings to build enriched representations that balance class-level consistency with instance-level sensitivity, leading to better generalization to unseen classes. Our approach not only achieves state-of-the-art performance on 11 few-shot recognition benchmarks, but also offers a plug-and-play solution that enhances interpretability through SAE.

목차 (Table of Contents)

• 1. Introduction 1
• 2. Related Works 5
• 2.1. Few-shot Learning 5
• 2.2. Prompt tuning 7
• 2.3. Sparse autoencoder 8
• 3. Methodology 9
• 3.1. Preliminary 9
• 3.2. Pre-training 12
• 3.3. Disentangled Instance-Class knowlEdge prompt tuning via SAE for Vision-Language Models 14
• 4. Experiment 19
• 4.1. Experiment Setup 19
• 4.2. Base-to-Novel Generalization 20
• 4.3. Cross-Dataset Evaluation 23
• 4.4. Disentanglement 24
• 4.5. Ablative Analysis 27
• 5. Conclusion 29