Robust spatiotemporal transformers for medical analysis using imaging and clinical data

다국어 초록 (Multilingual Abstract)

Recent advancements in medical image analysis have highlighted the growing potential of transformer-based architectures in modeling complex neuroimaging data. However, existing methods often struggle to effectively capture the high-dimensional spatial, temporal, and longitudinal patterns inherent in brain MRI and fMRI scans. Furthermore, limited progress has been made in seamlessly integrating such imaging modalities with structured clinical data for comprehensive patient profiling.

This thesis presents a unified transformer framework for neuroimaging analysis, covering volumetric MRI (3D), time-series fMRI (4D), and longitudinal fMRI (5D). A series of CrossFormer models are proposed to extract hierarchical spatial, temporal, and longitudinal representations via multi-scale tokenization and dual-range attention mechanisms. Additionally, two multimodal fusion architectures are introduced to integrate structured clinical phenotype data with both 3D and 4D neuroimaging features using intermediate fusion and cross-attentional interaction.

Extensive experiments across public datasets demonstrate the effectiveness of the proposed methods. The 3D CrossFormer achieved 94.6% accuracy on the ADNI dataset for Alzheimer’s classification. On the ABIDE dataset, the multimodal 3D MRI–clinical fusion model reached 93.2% accuracy in autism diagnosis. On the ADHD-200 dataset, the 4D fMRI CrossFormer obtained 91.2% accuracy, and the multimodal 4D fusion model incorporating clinical features achieved 94.5% accuracy. Furthermore, the longitudinal 5D fMRI CrossFormer attained 92.3% accuracy, demonstrating the benefit of modeling temporal evolution across clinical visits.

Beyond predictive performance, the models produce interpretable attention visualizations that highlight salient brain regions and modality interactions. This thesis contributes a generalizable, explainable, and scalable transformer framework for advanced multimodal neuroimaging analysis.

국문 초록 (Abstract)

본 논문은 다양한 신경영상 데이터에서 공간적 및 시간적 특징을 효과적으로 포 착하기 위한 계층적 Transformer 기반의 CrossFormer 모델 시리즈를 제안한다. 먼 저, 3D CrossFormer는 다중 스케일 토크나이제이션과 단기-장기 어텐션 메커니즘 을 활용하여 3차원 MIRI 데이터에서 88.9%의 정확도로 높은 성능을 달성하였다. 이 어서, 4D AMRI 데이터를 대상으로 하는 4D CrossFormer는 시간 종속성에 대한 정 교한 모델링을 통해 92.3%의 정확도를 기록하며 추가적인 성능 향상을 보여주었다. 또한, 5D CrossFormer는 종단적 AMRI 시계열을 통합하여 시간-공간- 세션 간 관계 를 효과적으로 학습하였고, AD 진단 및 진행 예측 과제에서 93.1%의 정확도를 달성하였다.
본 논문은 구조화된 임상 특성과 영상 표현을 결합하는 중간 융합 아키텍처도 제안한다. 3D MRI와 임상 데이터의 융합 모델은 89.6%의 정확도를 달성하였고, 4D fMRI와 임상 정보를 통합한 다중모달 모델은 93.9%의 정확도를 보이며 단일 모달대비 우수한 성능을 입증하였다. 본 연구는 다양한 데이터 차원과 모달리티에서Transformer 아키텍처의 일반화 가능성과 확장성을 입증하며, 다차원 의료영상 분석의 새로운 방향을 제시한다.

목차 (Table of Contents)

• 1 Introduction 1
• 1.1 Motivation and Background 1
• 1.2 Spatiotemporal Modeling in Medical Imaging 3
• 1.3 Multimodal Integration with Clinical Knowledge 6
• 1.4 Research Objectives and Contributions 7
• 1.5 Thesis Structure 9
• 2 Related Works 11
• 2.1 Structural MRI Analysis with Deep Learning 11
• 2.2 Functional MRI Modeling and Spatiotemporal Transformers 13
• 2.3 Longitudinal Neuroimaging and Disease Progression Modeling 15
• 2.4 Multimodal Fusion of Imaging and Clinical Features 16
• 3 Methods 19
• 3.1 3D MRI CrossFormer Architecture 20
• 3.2 4D fMRI CrossFormer Architecture 23
• 3.3 5D fMRI CrossFormer Architecture 30
• 3.4 Multimodal Architecture of MRI and Clinical Data 42
• 3.5 Multimodal Architecture of fMRI and Clinical Data 47
• 4 Experiments 62
• 4.1 Experiment Settings 63
• 4.2 Experiment Results of CrossFormer Architectures 69
• 4.3 Experiment Results of Multimodal Architectures 91
• 4.4 Overall Performance Comparison on AD Dataset 109
• 5 Conclusion 113
• 5.1 Summary of Contributions 113
• 5.2 Limitations and Challenges 114
• 5.3 Future Directions 116
• References 117
• 논문요약 135