Inversion of Deep Generative Models: Structural and Optimization-based Approaches = 생성형 모델의 역변환: 구조적 그리고 최적화 기반 접근

다국어 초록 (Multilingual Abstract)

This thesis presents a comprehensive study on the inversion problem across major classes of deep generative models (DGMs), including normalizing flows (NFs), diffusion models (DMs), and variational autoencoders (VAEs). The investigation is structured around two primary approaches: structural methods for NFs and optimization-based techniques for DMs and VAEs. For NFs, the thesis identifies key issues in inverse stability, particularly in conditional scenarios, and proposes structurally modified coupling layers to enhance robustness. For DMs, a numerically exact inversion method based on backward Euler discretization is introduced, allowing accurate reconstruction even when using high-order solvers. In the case of VAEs, with a focus on latent diffusion models, a novel gradient-free decoder inversion algorithm is proposed to address the inefficiency of conventional optimization-based methods. The proposed solutions are supported by theoretical analysis and practical experiments across diverse tasks. Overall, this work contributes model-specific inversion strategies that improve the interpretability, controllability, and applicability of generative models, while also offering a foundation for future research on alternating inference between encoder and decoder.

국문 초록 (Abstract)

본 논문은 정규화 흐름, 확산 모델, 변분 오토인코더를 포함한 주요 딥 생성형 모델에 대한 역변환 문제를 종합적으로 다룬다. 본 연구는 정규화 흐름에 대한 구조 기반 접근과 확산 모델 및 변분 오토인코더에 대한 최적화 기반 접근이라는 두 가지 주요 방법론을 중심으로 구성된다. 정규화 흐름의 경우, 특히 조건부 생성 상황에서 역변환의 불안정성 문제를 규명하고, 이를 개선하기 위한 구조적으로 수정된 커플링 계층을 제안한다. 확산 모델에 대해서는 고차 방정식 해법에서도 정확한 복원이 가능하도록, 후진 오일러 이산화를 기반으로 한 수치적 역변환 기법을 고안하였다. 변분 오토인코더의 경우, 특히 최신 Latent 확산 모델에 초점을 맞추어, 기존의 gradient 기반 최적화 방식의 비효율성을 해결하기 위한 새로운 gradient-free 디코더 역변환 알고리즘을 제안한다. 제안된 방법들은 이론적 분석과 다양한 응용 실험을 통해 검증되었다. 본 연구는 생성 모델의 해석 가능성, 제어 가능성, 응용 가능성을 향상시키는 모델별 역변환 전략을 제시하며, 향후 인코더–디코더 반복 추론에 기반한 연구의 기반을 제공한다.

목차 (Table of Contents)

• 1 Introduction 1
• 1.1 Introduction 1
• 1.2 Generative Models 2
• 1.2.1 Definition and Objective 2
• 1.2.2 Key Types of DGMs 2
• 1.2.3 Application Benefits of Effective Inversion 4
• 1.3 Problem Formulation: Inversion of Generative Models 6
• 1.3.1 GAN Inversion 7
• 1.3.2 Inversion in NF, DM, and VAE 7
• 1.4 Organization of Thesis 10
• I Inversion of Normalizing Flows 15
• 2 On the Robustness of Normalizing Flows for Inverse Problems in Imaging 17
• 2.1 Introduction 17
• 2.2 Preliminaries 21
• 2.2.1 Conditional normalizing flow 21
• 2.2.2 Coupling transformations 22
• 2.2.3 Conditional NFs for inverse problems 23
• 2.3 On the Robustness of Conditional NFs 24
• 2.3.1 Exploding inverses in unconditional NFs 25
• 2.3.2 Exploding inverse in conditional NFs 27
• 2.3.3 OOD conditional inputs for conditional NFs 31
• 2.3.4 On how to avoid exploding inverse 32
• 2.4 Experimental Results 35
• 2.4.1 2D toy experiment 35
• 2.4.2 Super-resolution space generation 36
• 2.4.3 Low-light image enhancement 38
• 2.5 Discussion 40
• 2.6 Conclusion 41
• 3 Neural Diffeomorphic Non-uniform B-spline Flows 43
• 3.1 Introduction 43
• 3.2 Related Works 46
• 3.3 Diffeomorphic Non-uniform B-spline Flows 48
• 3.3.1 Definition for Flow Models 48
• 3.3.2 Sufficient Conditions for Diffeomorphism 50
• 3.3.3 Existence of an Analytic Inverse 52
• 3.3.4 Definitions on Various Domains 53
• 3.4 Experiments 58
• 3.4.1 Illustrative Toy Example 58
• 3.4.2 Boltzmann Generator Training by Force Matching 60
• 3.4.3 Dynamics Simulation by Density Estimation 63
• 3.4.4 Runtime Comparison 64
• 3.5 Discussion 65
• 3.6 Conclusion 66
• II Inversion of Diffusion Models 69
• 4 On Exact Inversion of DPM-Solvers 71
• 4.1 Introduction 71
• 4.2 Related Work 75
• 4.3 Background 77
• 4.3.1 Fast Sampling in DPM 77
• 4.3.2 Nave DDIM inversion 79
• 4.4 Proposed Method 79
• 4.4.1 Exact Inversion of DDIM 79
• 4.4.2 Exact Inversion of High-order DPM-Solvers 82
• 4.5 Experiments 83
• 4.5.1 Reconstruction 83
• 4.5.2 Application: Tree-ring watermark 85
• 4.5.3 Application: Background-preserving editing 87
• 4.6 Conclusion 89
• III Inversion of Variational Autoencoders 93
• 5 Gradient-free Decoder Inversion in Latent Diffusion Models 95
• 5.1 Introduction 95
• 5.2 Backgrounds 98
• 5.2.1 Latent diffusion models (LDMs) 98
• 5.2.2 Optimization-based GAN inversion 99
• 5.2.3 Gradient-based decoder inversion in LDMs 100
• 5.3 Gradient-free decoder inversion in LDMs 101
• 5.3.1 Motivation 101
• 5.3.2 Convergence analysis on forward step method 102
• 5.3.3 Convergence analysis on momentum for acceleration 103
• 5.3.4 Validation of the assumption 105
• 5.4 Experiments with practical optimization techniques 108
• 5.5 Application: Tree-rings watermarking for image generation 109
• 5.6 Discussion 110
• 5.6.1 Does the β-cocoercivity hold also when using Adam Yes. 110
• 5.6.2 Why this method has not been proposed in GAN inversion studies 111
• 5.7 Conclusion 112
• 6 Conclusion 115
• A Application of Robust Normalizing Flow: Adaptive Selection of Sampling-Reconstruction in Fourier Compressed Sensing 119
• A.1 Introduction 119
• A.2 Related Works 123
• A.2.1 Fourier compressed sensing 123
• A.2.2 Joint optimization of sampling-reconstruction 124
• A.2.3 Adaptive sampling 125
• A.2.4 Super-resolution space generation 126
• A.3 Proposed methods 127
• A.3.1 Adaptive selection of sampling-reconstruction 127
• A.3.2 How to and what to adaptively select 128
• A.4 Experiments 131
• A.4.1 Fourier CS face reconstruction 132
• A.4.2 Multi-coil CS-MRI reconstruction 135
• A.5 Discussion 137
• A.6 Conclusion 139