Hadoop 클러스터를 위한 모니터의 설계 및 구현

금태훈(Tae Hoon Keum),이원주(Won Joo Lee),전창호(Chang Ho Jeon) 대한전자공학회 2012 電子工學會論文誌-CI (Computer and Information) Vol.49 No.1

In this paper, we propose a new monitor for collecting job information from Hadoop clusters in real time. This monitor is made of two programs called Collector and Agent. Agent collects Hadoop cluster's node information and job information, and Collector analyzes the collected information and saves it in a database. Also, Collector was placed in a new node outside the Hadoop cluster so that it does not affect Hadoop's work and will not cause overload. When the proposed monitor was implemented and applied, the testbed cluster was able to detect the occurrence of dead nodes immediately. In addition, we were able to find Hadoop jobs which were inefficient and when we modified such jobs to further enhance the performance of Hadoop. 본 논문에서는 Hadoop 클러스터의 노드 정보와 작업 정보를 실시간으로 수집할 수 있는 새로운 모니터를 제안한다. 이 모니터는 Hadoop클러스터의 노드 정보와 작업 정보를 수집하는 Agent, 수집된 정보를 분석하고 데이터베이스에 저장하는 Collector로 구성된다. 또한 Collector를 Hadoop 클러스터에 참여하지 않은 새로운 노드에 위치시킴으로써 분석과정에서 발생하는 오버헤드로 인한 Hadoop의 작업지연을 제거한다. 제안한 모니터를 구현하고 실험적 클러스터에 적용함으로써, dead 노드의 발생을 실시간으로 파악할 수 있었다. 또한, Hadoop의 작업수행 과정에서 비효율적인 과정을 발견하고 개선함으로써 작업수행 시간을 단축시킬 수 있었다.

An Empirical Performance Analysis on Hadoop via Optimizing the Network Heartbeat Period

( Jaehwan Lee ),( June Choi ),( Hongchan Roh ),( Ji Sun Shin ) 한국인터넷정보학회 2018 KSII Transactions on Internet and Information Syst Vol.12 No.11

To support a large-scale Hadoop cluster, Hadoop heartbeat messages are designed to deliver the significant messages, including task scheduling and completion messages, via piggybacking to reduce the number of messages received by the NameNode. Although Hadoop is designed and optimized for high-throughput computing via batch processing, the real-time processing of large amounts of data in Hadoop is increasingly important. This paper evaluates Hadoop’s performance and costs when the heartbeat period is controlled to support latency sensitive applications. Through an empirical study based on Hadoop 2.0 (YARN) [1] architecture, we improve Hadoop’s I/O performance as well as application performance by up to 13 percent compared to the default configuration. We offer a guideline that predicts the performance, costs and limitations of the total system by controlling the heartbeat period using simple equations. We show that Hive performance can be improved by tuning Hadoop’s heartbeat periods through extensive experiments.

빅데이터의 반복적인 연산 작업을 지원하기 위한 Hadoop 기반 순환처리 시스템

홍승태,윤민,박경석,임채덕,장재우 한국정보과학회 2016 데이타베이스 연구 Vol.32 No.1

apReduce framework, have been actively done. Meanwhile, most of the big data analysis applications, e.g., genome data analysis, are required to do the same Map and Reduce functions repeatedly. However, Hadoop is inefficient for iterative data processing applications because it has a non-iterative processing structure. To solve this problem, we, in this paper, propose a Hadoop-based iterative processing system for supporting the repetitive computations of big data. In the proposed system, we first propose an iterative job scheduling technique for managing the iterative MapReduce jobs. Secondly, we propose an invariant data caching mechanism for reducing the I/O costs of data accesses. Thirdly, we propose a stopping condition check mechanism for preventing unnecessary computation. Fourthly, we propose an iterative resource scheduling technique for efficiently managing the resources of a Hadoop cluster. Finally, we show the performance superiority of the proposed system by comparing it with the existing Hadoop-based systems. 최근 빅데이터의 효율적인 분석을 위하여, 대표적인 MapReduce 프레임워크인 Hadoop에 대한 연구가 활발히 이루어지고 있다. 한편, 유전체 데이터 분석과 같이, 대부분의 빅데이터 분석 응용은 동일한 Map과Reduce 함수의 반복적인 수행을 요구한다. 그러나 Hadoop은 비순환처리 구조를 가지고 있기 때문에, 순환처리 응용에 비효율적인 문제점이 존재한다. 따라서 본 논문에서는 빅데이터의 반복적인 연산 작업을 지원하기위한 Hadoop 기반 순환처리 시스템을 제안한다. 제안하는 시스템은, 첫째, 반복적인 MapReduce job을 관리하기 위해 순환처리 job 스케줄링 기법을 제안한다. 둘째, 데이터 입출력 비용을 감소시키기 위해 불변 데이터캐싱 기법을 제안한다. 셋째, 불필요한 연산을 방지하기 위해 종료조건 검사 기법을 제안한다. 넷째, Hadoop 클러스터 자원의 효율적인 관리를 위해 순환처리 자원 관리 기법을 제안한다. 마지막으로, 기존 하둡 기반시스템과의 비교를 통해 제안하는 시스템의 성능 우수성을 보인다.

GPUedHadoop : Hadoop as Parallel Processing Framework for GPU Cluster

Bongen Gu,Seokil Song,Yoonsik Kwak 한국정보기술학회 2016 한국정보기술학회논문지 Vol.14 No.6

Many research groups try to use GPGPU to enhance the performance of Hadoop. In this paper, we propose new approach to enhance the performance of Hadoop Map task and Combiner by using GPGPU on Hadoop Cluster. Our approach is that the whole HDFS block called split is passed to Map task for GPU processing. And then, the result of Mapper enabling GPU processing is also passed to Combiner for GPU processing. In other words, accelerated steps via GPU are Mapper and Combiner in Hadoop. GPU-enabled Hadoop adopting our approach has the same characteristics as native Hadoop, and additionally high performance feature. To show that our approach is effective to enhance the performance of Hadoop by using GPU, we experiment on GPU-accelerated Hadoop. Our experimental results show that speedup factor of our approach is between 3.27 and 4.19. So, we can conclude that our approach for GPU-enabled Hadoop is effective to enhance the performance.

Research on the Performance Optimization of Hadoop in Big Data Environment

Jia Min-Zheng 보안공학연구지원센터 2015 International Journal of Database Theory and Appli Vol.8 No.5

In the age of Internet, the data transmission and storage got rapid progress, however, data processing and information extraction is still exist many problems to solve. Under the condition of so much data, processing data, get useful information; In cloud computing, big data environment to adopt the method of distributed computing, such a large complex networks, however, requires a simulation environment, for comparison and optimization platform, it can save development costs. Hadoop can evaluate the performance of distributed cloud computing platform, so the Hadoop performance directly affects the evaluation on the performance of the big data cloud computing, which fully show the importance of performance of Hadoop. Algorithm is improved based on Hadoop platform, using the particle swarm optimization algorithm improved the calculation and implementation of the Hadoop platform, so as to improve its ability to execute and compute, the calculation results and analysis show that the proposed scheme is effective.

Rhipe를 활용한 빅데이터 처리 및 분석

고영준,김진석 한국데이터정보과학회 2013 한국데이터정보과학회지 Vol.24 No.5

The Hadoop system was developed by the Apache foundation based on GFS and MapReduce technologies of Google. Many modern systems for managing and processing the big data have been developing based on the Hadoop because the Hadoop was designed for scalability and distributed computing. The R software has been considered as a well-suited analytic tool in the Hadoop based systems because the R is flexible to other languages and has many libraries for complex analyses. We introduced Rhipe which is a R package supporting MapReduce programming easily under the Hadoop system, and implemented a MapReduce program using Rhipe for multiple regression especially. In addition, we compared the computing speeds of our program with the other packages (ff and bigmemory) for processing the large data. The simulation results showed that our program was more fast than ff and bigmemory as the size of data increases. 최근 Hadoop은 빅데이터의 저장, 처리 및 분석을 위한 표준시스템으로 인식되고 있으며, 많은 빅데이터 관련 시스템들이 Hadoop에 기반하여 구축되고 있다. 또한 R은 다른 소프트웨어와의 연동이 쉽고 다양한 분석 라이브러리들을 탑재하고 있어서 Hadoop 환경하에서 빅데이터의 분석을 위한 공통 분석 플렛폼으로 여겨지고 있다. 본 논문에서는 Hadoop 환경에서 분산 데이터 처리를 위한 R패키지인 Rhipe를 소개하고 빅데이터를 이용한 병렬 다중회귀분석을 위해 MapReduce 프로그램을 작성하는 방법을 예시하였다. 또한 시뮬레이션을 통해 기존의 대용량처리를 위한 R 분석패키지인 ff와 bigmemory와의 연산속도를 비교하였으며, 데이터의 크기가 커짐에 따라 Rhipe를 이용한 MapReduce 프로그램의 계산속도가 ff와 bigmemory에 비해 우수함을 확인하였다.

GPU 클러스터에서 Hadoop Map/Reduce 구현 방안

구본근(Bon-Gen Gu) 한국정보기술학회 2012 Proceedings of KIIT Conference Vol.2012 No.5

본 논문에서는 GPU 클러스터에 Hadoop 플랫폼을 적용할 수 있는 방안을 제시한다. 이를 위한 주요 방법으로는 HDFS 블록을 레코드 단위로 나누어 처리하는 것이 아니라 블록 전체를 GPGPU 메모리에 적재할 수 있는 입력 스플릿을 설계, 구현하는 것이다. 또 Map 태스크에서 JNI를 통해 GPU 코드를 호출하고, 처리 결과를 Reduce 태스크에서 사용할 수 있도록 Map 태스크 수행 결과를 병합하는 것을 구현하는 것이다. 제안된 방법을 통해 GPU 클러스터 상에 Hadoop이 운용된다면 병렬 프로그램 작성의 programmability가 향상될 것으로 예상된다. Here, we propose the way for implementing Hadoop Map/Reduce framework on GPU cluster. In our way, the whole HDFS block is loaded on the GPGPU"s memory via new input split module implemented in future. GPU code is called by Map task via JNI, and the results returned by GPU code are merged to be used in Reduce task. If Hadoop is operated on GPU cluster via our way, the programmability for creating parallel applications is enhanced.

Data Analysis Technique for Massive Spatial Data Using Hadoop

Minwuk Jeon,Byoung-Woo Oh 보안공학연구지원센터 2016 International Journal of Database Theory and Appli Vol.9 No.8

The spatial data set has much useful information, but the amount of volume is massive and the type is complex. It makes hard to analyze the spatial data. There are software tools for general data. Hadoop is one of the tools to process the big data. Hadoop can be used to analyze the large amount of spatial data. This paper proposed a data analysis technique for massive spatial data using Hadoop. We extend the grid based clustering algorithm to use Hadoop. The grid based clustering algorithm makes clusters with cells. Each cell has a number that counts contained objects. Only the cells who had the sufficient population can be join in clusters. The other cells ignored as noise. This paper proposed to enhance performance using Hadoop. In order to evaluate the enhancement of performance, the execution time is measured and compared. As the result, the proposed algorithm is 1.8 times faster than the original grid based clustering algorithm.

하둡에서 데이터 접근 제어 설계 및 구현

김희주,손시운,길명선,문양세 한국정보과학회 2014 데이타베이스 연구 Vol.30 No.2

Many projects have been developed integrating the Hadoop package, a current issue in the research area. Access control technology is one of the leading projects that the research area is focusing on. The access control technology has become necessary, in order to satisfy the needs of the data owners and users, especially with the data rapidly increasing due to internet developments and the growth of smart device users. In this paper, we propose a new data access control framework based on Hadoop Distributed File System (HDFS). The proposed method is composed of the new metadata storage module to manage user's file/directory authority information and the access management module which enables to check the authority of the user permission to create/modify/delete a file. With the design of a data access control framework using the two modules, it allows to provide access control function to users who are using platform of bigdata. The proposed framework is designed inside the Hadoop so that no additional installation is needed from the existing platform, expecting higher utilization in the future. Next, the metadata is changed from the XML format to a database format using the HBase. This method is based on the observation that the XML base metadata structure is difficult to be used in the bigdata structure which consist large amount of data, and for security reasons as it does not save the file inside the Hadoop. As for this, we first design the HBase table structure, and ensure that this works within the Hadoop. Finally, by designing and implementing the portal that can be actually utilized, affirmation is made whether the access control function works correctly within the proposed framework. 최근 여러 분야에서 하둡(Hadoop) 환경의 활용이 증가하면서, 데이터 소유자 및 사용자의 니즈에 의해 접근제어 기술의 필요성이 대두되고 있다. 따라서, 본 논문에서는 HDFS(Hadoop Distributed File System, 하둡분산 파일 시스템) 기반의 새로운 데이터 접근 제어 프레임워크를 제안하고자 한다. 제안하는 방법은 파일/디렉터리에 대한 사용자 권한 정보를 관리하는 “메타데이터 저장 모듈”과 파일 접근 권한 관리 및 사용자 권한 확인기능을 갖는 “접근 관리 모듈”로 구성된다. 또한, 제안한 프레임워크는 기존 플랫폼에 추가적인 설치가 필요 없도록 하둡 내부에 설계하여 향후 활용도가 높을 것이라 기대된다. 제안한 프레임워크는 먼저 XML 기반 메타데이터로 프레임워크의 정상 동작을 확인하고, 이후 메타데이터를 HBase 기반으로 관리 가능하도록 확장한다. 이는XML 기반 구조가 빅데이터 환경에서는 사용하기 어려우며, 단일 XML 파일에서 모든 권한 정보를 관리하면 보안상 위험이 있다는 관찰에 기반한다. 이러한 전체 모듈 구성을 바탕으로 접근 관리 포탈을 설계 및 구현하여 제안한프레임워크의 접근 제어 기능이 정상적으로 동작함을 확인한다.

Hadoop Parameter 조정을 통한 Hadoop MapReduce 성능 최적화 분석 연구

지경엽(Keungyeup Ji),권영미(Youngmi Kwon) 한국정보기술학회 2021 한국정보기술학회논문지 Vol.19 No.6

The purpose of this paper is to understand the performance optimization environment through parameter adjustment of the Hadoop environment as the most efficient way to solve the inefficiency of speed, which is the disadvantage of the Hadoop framework that processes large amounts of data based on distributed processing and virtualization. To this end, we derived optimization of the HDFS and MapReduce performance improvement that support Hadoop. In this paper, through this test, in order to utilize the Hadoop framework in an optimal state under a given resource condition, it is considered efficient to perform data processing after deriving an appropriate configuration environment through Hadoop parameter adjustment.