Reliable and Interpretable Evaluation in Deep Representational Models

다국어 초록 (Multilingual Abstract)

Recent advances in deep generative models in computer vision have extended their capabilities from image generation to diverse domains such as video and 3D object generations. What has driven these advancements at their core is the development of evaluation metrics that are reliable and accurate. These metrics assess generative models from a human perceptual perspective, measuring how closely the gen- erated data resembles real-world data and effectively highlighting their differences. This thesis inves- tigates recent advances in evaluation metrics by examining the key contributions of Article 1, Article 2, and Article 3. In addition, it identifies open challenges in evaluation that remain critical for the development of more powerful and reliable deep generative models. Article 1 introduces a novel evaluation metric for image generative models that measures the level of realism along two key aspects: fidelity and diversity. Existing metrics typically estimate the distributions of real and generated data in model embedding spaces that reflect human perception, and compute scores by comparing these distributions. However, generative models that are not properly trained often produce noisy data, and in the presence of such noise, existing metrics are unable to provide reliable and accurate evaluations. To address this issue, this work proposes a robust evaluation approach by estimating statistically and topologically significant supports for both real and generated data. This distribution estimation method is sensitive to subtle variations in the data distribution and provides more accurate and reliable evaluation results, even in the presence of noise. Article 2 introduces a novel evaluation metric for video generative models that measures realism along three aspects: fidelity, diversity, and temporal naturalness. Existing video metrics have largely relied on techniques developed for image generation models, which often fail to capture the temporal characteristics inherent in video data, resulting in incomplete or unreliable evaluations. To address this limitation, this work leverages the observation that frame-wise changes in typical videos exhibit amplitude distributions following a power law in the Fourier domain. By estimating this power law distribution, the proposed metric quantitatively measures the deviation of generated videos from the natural distribution, providing the first principled evaluation of temporal consistency in video generation. Article 3 proposes a benchmark that enables comparison between object recognition models and humans, and allows model analysis from a human visual perspective. The existing benchmark, using stylized images that blend shape and texture within a single image, suggests that humans primarily rely on shape, whereas models focus on texture. However, this prior work suffers from several limitations: (1) it does not utilize data representing pure shape and pure texture, (2) it does not consider images in which shape and texture are present in equal proportion (50:50), and (3) it employs evaluation measures that are not well-suited for model analysis and comparison. To address these limitations, Article 3 generates disentangled datasets that contain pure shape and texture cues and proposes a new metric that enables reliable and precise evaluation of models. This benchmark provides a clear and unbiased assessment of current object recognition models, enabling accurate measurement of how closely their reliance on shape and texture aligns with human perception.

목차 (Table of Contents)

• Abstract 1
• List of Tables 8
• List of Figures 13
• Chapter. 1 Introduction 14
• Chapter. 2 Background 17
• 2.1 Preliminaries on Generative Models 17
• 2.1.1 Generative Adversarial Networks (GAN) 17
• 2.1.2 Diffusion Models 17
• 2.1.3 Generative Models as Deep Representation Models 18
• 2.2 Embedding Networks for Generative Model Evaluation 18
• 2.2.1 Supervised Learning: Classification Models 18
• 2.2.2 Self-supervised Learning: Contrastive Learning Models 19
• 2.3 Evaluation Metrics for Generative Models 19
• 2.3.1 Single Score Metrics 19
• 2.3.2 Metrics for Detailed Model Analysis 20
• Chapter. 3 Evaluation of Image Generative Models 24
• 3.1 Introduction 25
• 3.2 Background 26
• 3.3 Methodology 28
• 3.3.1 Topological precision and recall (TopP&R) 28
• 3.3.2 Removing Statistical and Topological Noise using Bootstrap Confidence Bands . 29
• 3.3.3 Addressing the curse of dimensionality 29
• 3.4 Consistency with Robustness of TopP&R 30
• 3.5 Experiments 31
• 3.5.1 Sanity checks with toy data 32
• 3.5.2 Sanity check with real data 34
• 3.6 Discussion 36
• 3.7 Conclusions 36
• Chapter. 4 Evaluation of Video Generative Models 40
• 4.1 Introduction 41
• 4.2 Spatio-TempoRal Evaluation and Analysis Metric (STREAM) 42
• 4.2.1 Notation 42
• 4.2.2 Embedding network 43
• 4.2.3 Temporal modeling of image embeddings through FFT 43
• 4.2.4 STREAM-T: evaluating the temporal flow of videos 43
• 4.2.5 STREAM-S: evaluating the spatial quality of videos 45
• 4.3 Experiments 46
• 4.3.1 Toy data experiment 46
• 4.3.2 Real data experiment 49
• 4.4 Related Works 51
• 4.5 Conclusion 51
• Chapter. 5 Shape and Texture Bias Evaluation of Deep Neural Networks 54
• 5.1 Introduction 55
• 5.2 Preliminaries: Cue-conflict Benchmark 56
• 5.2.1 Where Cue-conflict Falls Short? 57
• 5.3 Methodology 60
• 5.3.1 Shape and Texture Cue Construction 60
• 5.3.2 Model Comparisons with Redefined Bias 61
• 5.4 Validating REFINED-BIAS Benchmark 62
• 5.4.1 Learning Strategies and Hypothesis 62
• 5.4.2 Validity of REFINED-BIAS Benchmark 63
• 5.5 Findings from Empirical Analysis 64
• 5.5.1 Model Architectures 65
• 5.5.2 Natural Bias Trade-off in CNN variants 65
• 5.5.3 How ViT Design Influences Bias 65
• 5.5.4 Consistent Bias-Performance Analysis 66
• 5.6 Discussion 66
• 5.7 Conclusion 66
• Chapter. 6 Discussion and Conclusion 71
• Chapter. 7 Future Works 73
• Chapter. A Appendix to First Article 74
• A.1 More Background on Topological Data Analysis 74
• A.1.1 Persistent homology 74
• A.1.2 Statistical inference of persistent homology 74
• A.2 Johnson-Lindenstrauss Lemma 75
• A.3 Denoising Topological Features from Outliers 75
• A.4 Assumptions on Distributions and Kernels 76
• A.5 Details and Proofs for Section 3.4 78
• A.6 Related Work 89
• A.6.1 Improved Precision & Recall (P&R) 89
• A.6.2 Density & Coverage (D&C) 90
• A.6.3 Geometric Evaluation of Data Representations (GCA) 90
• A.6.4 Manifold Topology Divergence (MTD) 91
• A.7 Philosophy of our Metric & Practical Scenarios 91
• A.7.1 Philosophy of our metric 91
• A.7.2 Practical scenarios 91
• A.7.3 Details of the noise framework in the experiments 92
• A.7.4 Limitations 93
• A.8 Experimental Details 93
• A.8.1 Implementation details of embedding 93
• A.8.2 Implementation details of confidence band estimator 94
• A.8.3 Choice of confidence level 94
• A.8.4 Estimation of bandwidth parameter 94
• A.8.5 Computational complexity 95
• A.8.6 Mean Hamming Distance 95
• A.8.7 Explicit values of bandwidth parameter 96
• A.9 Additional Experiments 98
• A.9.1 Survivability of minority sets in the long-tailed distribution 98
• A.9.2 Experiment on Non-IID perturbation with outlier removal methods 99
• A.9.3 Evaluating state-of-the-art generative models on the ImageNet 100
• A.9.4 Verification of random projection effect in generative model ranking 101
• A.9.5 Sensitiveness to noise intensity 101
• A.9.6 Verifying the effect of random projection to the noisy data 102
• A.9.7 Robustness of TopP&R with respect to random projection dimension 103
• A.9.8 Trucation trick 103
• A.9.9 Resolving fidelity and diversity 103
• A.9.10 Toy experiment of trade-off between fidelity and diversity 104
• Chapter. B Appendix to Second Article 107
• B.1 More Details on STREAM 107
• B.1.1 Moment Generating Function 107
• B.1.2 Why STREAM-S utilizes amplitude at frequency 0 107
• B.1.3 Validating STREAM against human judgment 107
• B.2 Ablation Studies 108
• B.2.1 Effect of using a video embedding network on STREAM-T 108
• B.2.2 How the data size affects STREAM? 109
• B.2.3 Effect of histogram bin size on STREAM 109
• B.2.4 How do video resolution and length affect STREAM? 110
• B.3 Qualitative Examples 112
• B.3.1 Example of temporal distortions using kinetics-600 dataset 112
• B.3.2 Qualitative examples of generative models that produce short videos 113
• B.3.3 Qualitative examples of long videos generated by the TATS-base model 114
• B.3.4 Qualitative examples of generative models that produce long videos 115
• B.4 Additional Experiments 116
• B.4.1 How STREAM behaves when the video is blurry? 116
• B.4.2 How does STREAM behave when the video is reversed or flipped? 116
• B.4.3 Comparison between STREAM and VIS 117
• Chapter. C Appendix to Third Article 119
• C.1 Details on REFINED-BIAS 119
• C.1.1 Category Selection Guided by Visual Cue Distinctiveness 119
• C.1.2 Psychophysical User Study Design and Examples 120
• C.1.3 Relative Task Difficulty in REFINED-BIAS 122
• C.1.4 Bias computation in cue-conflict 122
• C.1.5 Examples of Confounded Cues from Stylization 124
• C.1.6 Examples of Imbalanced Cue Information by Stylization 124
• C.2 Comparison of Model Architectures 125
• C.2.1 Details on Model Comparison across Different Architectures 125
• C.3 Comparison of Model Training Strategies 126
• C.3.1 Details on Model Comparison across Different Training Strategies 126
• C.3.2 Confidence Intervals for Models with Different Training Strategies 127
• C.3.3 Configurations of Models trained with Mixed Augmentations 128
• C.3.4 Visual Examples for Different Learning Strategies 128
• Acknowledgements 132