(A) new approach to metadata management in HDFS

다국어 초록 (Multilingual Abstract)

In this paper, we present the features of the HDFS, which stands for Hadoop Distributed File System. HDFS is a distributed user-level file system to manage storage resources across a cluster, and is designed for high availability and portability across heterogeneous hardware and software platforms [1]. HDFS consists of a single Namenode and multiple Datanodes. This paper points a bottleneck of HDFS in the Namenode while introducing the architecture of HDFS. This problem is from the Master-Worker structure of HDFS nodes and the Namenode is suffered from heavy loads from many HDFS clients and Datanodes. As a solution of reducing loads in the Namenode, we propose a new approach to managing metadata in HDFS: it is to provide HDFS clients a cache to save the mapping information of requested files and their blocks.

목차 (Table of Contents)

• 1 Introduction 8
• 2 Background 9
• 2.1 Hadoop and HDFS 9
• 2.1.1 What is Hadoop? 9
• 2.1.2 HDFS (Hadoop Distributed File System) 9
• 2.2 HDFS Architecture 9
• 2.2.1 A Namenode & Datanodes 10
• 2.2.2 Blocks 11
• 2.2.3 Data Flow in HDFS client 11
• 2.3 SPOF (Single Point of Failure) of HDFS 13
• 3. Client-side Metadata Cache 14
• 3.1 Overview 14
• 3.2 Architecture & Implementation 14
• 4. Experiments 16
• 4.1 Overview 16
• 4.1.1 Experiment Environment I 16
• 4.1.2 Experiment Environment II 17
• 4.2 Experiment Scenario 17
• 4.2 Experiment Result 18
• 5. Future Work 21
• 5.1 Overview 21
• 5.2 Haystack as Photo Storage in Facebook 21
• 5.4 What is Haystack? 21
• 5.4 Haystack Architecture and Mechanism when downloading files 21
• 6. Conclusion 23
• References 24