Toward Holistic Memory Safety Across the Software-Hardware Stack = 소프트웨어-하드웨어 전반의 메모리 안전성 연구

다국어 초록 (Multilingual Abstract)

Memory corruption vulnerabilities remain a persistent and critical threat to modern computing systems, particularly those built with memory-unsafe languages like C and C++. While numerous defense mechanisms have been proposed across software and hardware layers, attackers continue to exploit weaknesses in system design, implementation, and interfaces between components.

This dissertation presents a comprehensive study of modern memory safety challenges and introduces practical defenses that bridge the software-hardware boundary. We begin by addressing access control integrity in the Linux kernel through PeTAL, a data-flow integrity solution that ensures the correctness of kernel access control decisions. Leveraging ARM’s Pointer Authentication Code (PAC) and Memory Tagging Extension (MTE), PeTAL protects critical metadata from data-only attacks and complements existing control-flow integrity mechanisms, achieving strong security guarantees with practical performance overhead.

While hardware-assisted features like MTE are considered promising for memory safety, this work reveals their limitations by presenting TikTag, a microarchitectural sidechannel attack that leaks memory tags using speculative execution. TikTag breaks the core security assumption of MTE’s random tagging mechanism and demonstrates real-world exploitation in both user-space (Chrome browser) and kernel-space environments.

Finally, this dissertation tackles an emerging class of vulnerabilities stemming from incorrect virtual-to-physical memory mappings. We introduce DMCheck, a runtime mechanism to detect and prevent physical page use-after-free vulnerabilities in modern systems. By tracking the lifecycle of physical pages and ensuring mapping validity across diverse computing units (CPU, GPU, and IOMMU), DMCheck secures critical memory translation paths with minimal overhead.

Together, these contributions push the boundary of memory safety by demonstrating the feasibility and necessity of holistic protection strategies. By securing both software, hardware, and underlying software-to-hardware translation mechanisms, this dissertation highlights new directions toward building more resilient system in the presence of powerful adversaries.

국문 초록 (Abstract)

메모리 손상 취약점은 시스템 보안에서 가장 근본적이고 심각한 위협으로 남아 있다. 본 논문은 이러한 위협에 대응하기 위해 시스템 전반의 메모리 안전성을 강화하는 세 가지 접근 방안을 제시한다.

첫째, PeTAL 은 ARM 아키텍처의 Pointer Authentication Code (PAC)과 Memory Tagging Extension (MTE)를 결합하여, 커널 내 접근 제어 흐름에 대한 데이터 흐름 무결성을 보장하는 방어 기법을 제안한다.
둘째, TikTag 은 MTE 태그 정보를 누출할 수 있는 새로운 마이크로아키텍처 부채널 공격을 제시함으로써, MTE 기반 메모리 보호 기법이 부채널 공격에 취약할 수 있음을 실증적으로 보인다.
셋째, DMGuard 는 CPU와 GPU 등 다양한 컴퓨팅 유닛을 아우르는 페이지 생애주기 관리를 통해 물리 페이지 단위의 use-after-free 취약점을 효율적으로 탐지 하고 방어하는 기법을 제안한다.

이 세 가지 기술을 통해 본 논문은 소프트웨어와 하드웨어의 경계를 아우르는 공 격 표면을 체계적으로 분석하고, 실제 시스템에 적용 가능한 실용적이며 종합적인 메모리 보안 방안을 제시한다.

목차 (Table of Contents)

• Abstract i
• 1 Introduction 1
• 1.1 Introduction 1
• 2 PeTAL: Ensuring Access Control Integrity against Data-only Attacks on Linux 4
• 2.1 Introduction 4
• 2.2 Background 7
• 2.3 Access Control Systems and Attacks 8
• 2.3.1 Access Control Systems 8
• 2.3.2 Attacks on Access Control Systems 11
• 2.4 Threat Model 15
• 2.5 Enforcing Access Control Integrity 16
• 2.5.1 Identifying Protection Scope 17
• 2.5.2 Enforcing Two-Level Data-Flow Integrity 20
• 2.6 Implementation 27
• 2.7 Evaluation 32
• 2.7.1 Effectiveness of Protection Scope 32
• 2.7.2 Security Analysis 34
• 2.7.3 Concrete Attack Analysis 38
• 2.7.4 Runtime Overhead 39
• 2.8 Discussion 42
• 2.9 Related work 43
• 2.10 Conclusion 44
• 3 TikTag: Breaking ARM’s Memory Tagging Extension with Speculative Execution 45
• 3.1 Introduction 45
• 3.2 Background 48
• 3.2.1 Memory Tagging Extension 48
• 3.2.2 Speculative Execution Attack 49
• 3.3 Threat Model 50
• 3.4 Finding Tag Leakage Gadgets 50
• 3.4.1 Tag Leakage Template 51
• 3.4.2 Tag Leakage Fuzzing 52
• 3.5 TikTag Gadgets 54
• 3.5.1 TikTag-v1: Exploiting Speculation Shrinkage 54
• 3.5.2 TikTag-v2: Exploiting Store-to-Load Forwarding 61
• 3.6 Real-World Attacks 64
• 3.6.1 Attacking Chrome Browser 65
• 3.6.2 Attacking the Linux Kernel 69
• 3.7 Evaluation 75
• 3.7.1 Chrome Browser Tag Leakage 75
• 3.7.2 Linux Kernel Tag Leakage 79
• 3.8 Related work 81
• 3.9 Conclusion 83
• 4 DMGuard: Safeguarding Modern Kernels from Physical-Page Use-After-Free Vulnerabilities 84
• 4.1 Introduction 84
• 4.2 Motivation 87
• 4.2.1 Complexity in Page Management 88
• 4.2.2 Dangling Mapping Causing Page Use-After-Free 89
• 4.2.3 Real-World Example 91
• 4.3 DMGuard 92
• 4.3.1 Page Lifecycle as a State Machine 93
• 4.3.2 Twofold State Transition Tracking 95
• 4.3.3 Putting It All Together: Lockless Integration 100
• 4.4 Implementation 104
• 4.5 Evaluation 108
• 4.5.1 Evaluation Setup 108
• 4.5.2 Security Evaluation 110
• 4.5.3 Performance Evaluation 113
• 4.5.4 Robustness Evaluation 117
• 4.6 Discussion 117
• 4.7 Related work 119
• 4.8 Conclusion 120
• 5 Conclusion 121
• 초록 136
• 감사의 글 137