동적 분산병렬 하둡시스템 및 분산추론기에 응용한 서버가상화 빅데이터 플랫폼 = An elastic distributed parallel Hadoop system for bigdata platform and distributed inference engines

다국어 초록 (Multilingual Abstract)

Inference process generates additional triples from knowledge represented in RDF triples of semantic web technology. Tens of million of triples as an initial big data and the additionally inferred triples become a knowledge base for applications such as QA(question&answer) system. The inference engine requires more computing resources to process the triples generated while inferencing. The additional computing resources supplied by underlying resource pool in cloud computing can shorten the execution time. This paper addresses an algorithm to allocate the number of computing nodes "elastically" at runtime on Hadoop, depending on the size of knowledge data fed. The model proposed in this paper is composed of the layered architecture: the top layer for applications, the middle layer for distributed parallel inference engine to process the triples, and lower layer for elastic Hadoop and server visualization. System algorithms and test data are analyzed and discussed in this paper. The model hast the benefit that rich legacy Hadoop applications can be run faster on this system without any modification.

참고문헌 (Reference)

1 박정혁, "하둡과 맵리듀스" 한국데이터정보과학회 24 (24): 1013-1027, 2013

2 이완곤, "클라우드 컴퓨팅 환경에서의 대용량 RDFS 추론을 위한 분산 테이블 조인 기법" 한국정보과학회 41 (41): 674-685, 2014

3 이완곤, "분산 메모리 환경에서의 ABox 실체화 추론" 한국정보과학회 42 (42): 852-859, 2015

4 고영준, "Rhipe를 활용한 빅데이터 처리 및 분석" 한국데이터정보과학회 24 (24): 975-987, 2013

5 Zaharia, M., "Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing" USENIX Association Berkeley 2012

6 Agarwal, S., "Re-optimizing data-parallel computing" 2012

7 Pastorelli, M., "HFSP: Size-based scheduling for Hadoop" 2013

8 Verma, A., "Aria: Automatic resource inference and allocation for MapReduce environments" 2011

9 Song, D., "Annual report on distributed parallel infernce platform for large scale knowledge processing" IITP 2015

10 Antoniou, G., "A semantic web primer" The MIT Press 2012