하둡 기반 대규모 작업처리 프레임워크에서의 Adaptive Parallel Computability 기술 연구

김직수 한국방송∙미디어공학회 2019 방송공학회논문지 Vol.24 No.6

We have designed and implemented a new data processing framework called MOHA(Mtc On HAdoop) which can effectively support Many-Task Computing(MTC) applications in a YARN-based Hadoop platform. MTC applications can be composed of a very large number of computational tasks ranging from hundreds of thousands to millions of tasks, and each MTC application may have different resource usage patterns. Therefore, we have implemented MOHA-TaskExecutor(a pilot-job that executes real MTC application tasks)’s Adaptive Parallel Computability which can adaptively execute multiple tasks simultaneously, in order to improve the parallel computability of a YARN container and the overall system throughput. We have implemented multi-threaded version of TaskExecutor which can “independently and dynamically” adjust the number of concurrently running tasks, and in order to find the optimal number of concurrent tasks, we have employed Hill-Climbing algorithm.

하둡 기반 Many-Task Computing 응용 실행 최적화 기술 연구

최진영,강대철,김직수 한국정보과학회 2019 정보과학회 컴퓨팅의 실제 논문지 Vol.25 No.12

MTC (Many-Task Computing) is a new computing paradigm that efficiently processes a significantly large number of tasks with relatively short running times. MTC paradigm can be applied in various scientific domains, and in order to effectively support these MTC applications we must devise high-performance task dispatching mechanism, dynamic resource allocation & load balancing scheme, and support for another type of data-intensive workload (compared to existing Big Data). We have developed MOHA(Mtc On HAdoop) framework which can effectively support MTC applications on top of the Hadoop platform. In this study, we propose an optimized environment for MTC application execution by verifying the high-performance task dispatching capability of MOHA, and conducting a comparative analysis of the HDFS and Lustre parallel file system. MTC(Many-Task Computing)는 상대적으로 짧은 작업 실행시간을 지니는 대규모 작업들을 고성능으로 처리하고자 하는 새로운 컴퓨팅 패러다임이다. 다양한 과학 응용 분야에서 MTC 패러다임을 요구하고 있으며, 이러한 MTC 응용들을 효과적으로 지원하기 위해서는 기존 배치 스케줄러의 성능을 뛰어넘는 고성능 작업 배포, 동적인 자원 할당 및 로드 밸런싱, 기존 빅데이터 응용과는 또 다른 형태의 데이터 집약형 워크로드 지원 등과 같은 기능들을 필요로 한다. 이에 따라 본 연구팀에서는 하둡(Hadoop) 플랫폼에서 대용량의 태스크들을 효과적으로 처리할 수 있는 MOHA(Mtc On HAdoop) 프레임워크를 개발하여 고성능 태스크 배포 성능을 확인하고, MTC 응용을 위한 스토리지 최적화를 위해 하둡의 기본 데이터 스토리지인 HDFS와 슈퍼컴퓨팅 분야의 Lustre 병렬파일시스템의 성능을 비교/분석함으로써 MTC 응용 실행에 적합한 환경을 제안하고자 한다.

대규모 이종 메시지의 실제 응용을 통한 분산 메시징 시스템 활용 연구

이혁진,장현지,김직수 한국정보과학회 2022 정보과학회 컴퓨팅의 실제 논문지 Vol.28 No.5

Many-Task Computing(MTC) is a new computing paradigm comprising a very large number of tasks(hundreds of thousands or even more) with relatively short per task running time, and a large variance of execution times. We conducted a precedent study about leveraging distributed messaging systems to effectively support these challenging MTC applications with many tasks, and found that messages used for describing MTC tasks become “heterogeneous messages“ of which processing times vary widely. In this paper, we present comprehensive experimental results of heterogeneous message processing times of a real MTC application (drug repositioning), using three representative distributed messaging systems (ActiveMQ, RabbitMQ, Kafka). Through our comparative analysis, we suggest the effective distributed messaging system for processing large-scale heterogeneous messages. Many-Task Computing (MTC)은 새로운 컴퓨팅 패러다임으로서 주로 대규모의 태스크들로 구성(수십만 개 이상)되고, 상대적으로 짧은 실행 시간과 큰 작업 실행 편차 등의 특징들을 나타내고 있다. 본 연구팀에서는 이러한 대규모의 태스크들로 구성된 MTC 응용을 효과적으로 지원하기 위해 분산 메시징 시스템을 활용한 선행 연구를 수행하였고, MTC 응용의 특성상 메시지 처리 실행 시간의 편차가 나타날 수 있음에 기반하여 이를 “이종 메시지”라고 규정하였다. 본 논문에서는 이러한 이종 메시지들을 효율적으로 처리하기 위해 3개의 대표적인 분산 메시징 시스템(ActiveMQ, RabbitMQ, Kafka)들을 활용하여 실제 MTC 응용(신약 재창출)을 통한 이종 메시지 처리 실행 시간을 측정한 결과를 제시한다. 이들의 실행 시간 비교 분석을 통해 대규모 이종 메시지 처리에 적합한 분산 메시징 시스템을 제안하고자 한다.

Many-objective joint optimization for dependency-aware task offloading and service caching in mobile edge computing

Xiangyu Shi,Zhixia Zhang,Zhihua Cui,Xingjuan Cai 한국인터넷정보학회 2024 KSII Transactions on Internet and Information Syst Vol.18 No.5

Previous studies on joint optimization of computation offloading and service caching policies in Mobile Edge Computing (MEC) have often neglected the impact of dependency-aware subtasks, edge server resource constraints, and multiple users on policy formulation. To remedy this deficiency, this paper proposes a many-objective joint optimization dependency-aware task offloading and service caching model (MaJDTOSC). MaJDTOSC considers the impact of dependencies between subtasks on the joint optimization problem of task offloading and service caching in multi-user, resource-constrained MEC scenarios, and takes the task completion time, energy consumption, subtask hit rate, load variability, and storage resource utilization as optimization objectives. Meanwhile, in order to better solve MaJDTOSC, a many-objective evolutionary algorithm TSMSNSGAIII based on a three-stage mating selection strategy is proposed. Simulation results show that TSMSNSGAIII exhibits an excellent and stable performance in solving MaJDTOSC with different number of users setting and can converge faster. Therefore, it is believed that TSMSNSGAIII can provide appropriate sub-task offloading and service caching strategies in multi-user and resource-constrained MEC scenarios, which can greatly improve the system offloading efficiency and enhance the user experience.

A Bi-objective Game-based Task Scheduling Method in Cloud Computing Environment

Wanwan Guo,Mengkai Zhao,Zhihua Cui,Liping Xie 한국인터넷정보학회 2022 KSII Transactions on Internet and Information Syst Vol.16 No.11

The task scheduling problem has received a lot of attention in recent years as a crucial area for research in the cloud environment. However, due to the difference in objectives considered by service providers and users, it has become a major challenge to resolve the conflicting interests of service providers and users while both can still take into account their respective objectives. Therefore, the task scheduling problem as a bi-objective game problem is formulated first, and then a task scheduling model based on the bi-objective game (TSBOG) is constructed. In this model, energy consumption and resource utilization, which are of concern to the service provider, and cost and task completion rate, which are of concern to the user, are calculated simultaneously. Furthermore, a many-objective evolutionary algorithm based on a partitioned collaborative selection strategy (MaOEA-PCS) has been developed to solve the TSBOG. The MaOEA-PCS can find a balance between population convergence and diversity by partitioning the objective space and selecting the best converging individuals from each region into the next generation. To balance the players' multiple objectives, a crossover and mutation operator based on dynamic games is proposed and applied to MaPEA-PCS as a player's strategy update mechanism. Finally, through a series of experiments, not only the effectiveness of the model compared to a normal many-objective model is demonstrated, but also the performance of MaOEA-PCS and the validity of DGame.

Efficient and Parallel Data Processing and Resource Allocation in the Cloud by using Nephele’s Data Processing Framework

V.Saranya,S.Ramya,R.G. Suresh Kumar,T.Nalini 보안공학연구지원센터 2016 International Journal of Grid and Distributed Comp Vol.9 No.3

Cloud computing is a technology in which the Cloud Service Providers (CSP) provide many virtual servers to the users to store their information in the cloud. The faults occurring on the assignment and dismission of the virtual machines, the processing cost in the allocation of resources must also be considered. The parallel processing of the information on the virtual machines must be done effectively and in an efficient manner. A variety of systems were developed to facilitate Many Task Computing (MTC). These systems aim to hide the issues of parallelism and fault tolerant and they are used in many applications. In this paper, we introduced Nephele, a data processing framework to exploit dynamic resource provisioning offered by IaaS clouds. The performance evaluation of the virtual machines has been evaluated and the allocation and de-allocation of job tasks to the specific virtual machines has also been considered. A performance comparison with the well known data processing framework hadoop has been done. Thus this paper tells about the effective and efficient manner of processing the data by parallel processing and allocating the correct resources for the desired task. It also helps to reduce the cost of resource utilization by exploiting the dynamic resource utilization.

Towards a Next Generation Distributed Middleware System for Many-Task Computing

Jik-Soo Kim,Sangwan Kim,Seokkyoo Kim,Seoyoung Kim,Seungwoo Rho,Ok-Hwan Byeon,Soonwook Hwang 보안공학연구지원센터 2013 International Journal of Software Engineering and Vol.7 No.4

Design of a Large-scale Task Dispatching & Processing System based on Hadoop

김직수(Jik-Soo Kim),구엔 카오(Nguyen Cao),김서영(Seoyoung Kim),황순욱(Soonwook Hwang) Korean Institute of Information Scientists and Eng 2016 정보과학회논문지 Vol.43 No.6

A Case Study of Drug Repositioning Simulation based on Distributed Supercomputing Technology

김직수(Jik-Soo Kim),노승우(Seungwoo Rho),이민호(Minho Lee),김서영(Seoyoung Kim),김상완(Sangwan Kim),황순욱(Soonwook Hwang) Korean Institute of Information Scientists and Eng 2015 정보과학회논문지 Vol.42 No.1