하둡에서 태그를 이용한 빅데이터의 동적 분류에 관한 연구

다국어 초록 (Multilingual Abstract)

In last years, Hadoop has become a basic data processing infrastructure in the field of Big data processing. Due to design limitations of Hadoop, it is difficult to efficiently classify for real-time processing and identify information
of data. File in HDFS modification is not possible due to a feature of WORM-based. Also a very large block size is inefficient to process a small file. MapReduce to perform parallel analysis on a cluster is not suitable for real-time processing because the analysis proceeds around the batch processing.
In this paper, we proposes a method of real-time processing and dynamic classification of Big data using tags. The tagged data can be used to the real-time processing. In addition, it can assist batch processing of
MapReduce. The proposed method to lower memory usage of Hadoop name node, and when performed MapReduce, it was effective to reducing the number of mappers generated. In addition, it was confirmed that the tags that are useful for real-time processing and dynamic classification.

목차 (Table of Contents)

• Ⅰ. 서 론 ······························································································································· 1
• 1. 연구 배경 및 목적 ································································································ 1
• 2. 연구 내용 및 범위 ································································································ 3
• 3. 논문의 구성 ············································································································ 6
• Ⅱ. 관련연구 ························································································································· 7
• 1. 빅데이터 ·················································································································· 7
• 1) 빅데이터의 정의 ·························································································· 7
• 2) 전통적 도구의 빅데이터 처리 한계 ························································ 8
• 2. 분산시스템과 하둡 프레임워크 ·········································································· 9
• 1) 분산시스템 ···································································································· 9
• 2) 하둡 프레임워크 ························································································ 10
• 3. 실시간 처리와 하둡 ···························································································· 16
• 4. 스몰파일 ················································································································ 17
• 1) 스몰파일의 정의 ························································································ 17
• 2) 스몰파일의 하둡 성능저하 문제 ···························································· 18
• 3) 스몰파일 처리를 위한 관련기술 ···························································· 20
• Ⅲ. 태그를 이용한 실시간 처리 및 동적 분류 ························································ 24
• 1. 스몰파일 처리 ······································································································ 25
• 2. 태그를 이용한 실시간 처리 및 동적 분류 ···················································· 26
• Ⅳ. 설계 및 구현 ··············································································································· 29
• 1. 하둡 업무 처리과정 및 설계 요구사항 분석 ················································ 29
• 2. 클래스 설계 및 구현 ·························································································· 31
• Ⅴ. 실험 ······························································································································· 51
• 1. 빌드 및 태그를 통한 파일 분류 평가 ···························································· 51
• 2. 스몰파일 처리 확인 ···························································································· 62
• 1) HDFS 블록 사용량 평가 ········································································ 62
• 2) MapReduce의 Mapper 스케줄링 평가 ················································· 64
• Ⅵ. 결론 및 향후과제 ······································································································ 66