도커 기반의 실시간 데이터 연계 및 처리 환경을 고려한 빅데이터 관리 플랫폼 개발 = Development of Big-data Management Platform Considering Docker Based Real Time Data Connecting and Processing Environments

다국어 초록 (Multilingual Abstract)

Real-time access is required to handle continuous and unstructured data and should be flexible in management under dynamic state. Platform can be built to allow data collection, storage, and processing from local-server or multi-server. Although the former centralize method is easy to control, it creates an overload problem because it proceeds all the processing in one unit, and the latter distributed method performs parallel processing, so it is fast to respond and can easily scale system capacity, but the design is complex. This paper provides data collection and processing on one platform to derive significant insights from various data held by an enterprise or agency in the latter manner, which is intuitively available on dashboards and utilizes Spark to improve distributed processing performance. All service utilize dockers to distribute and management. The data used in this study was 100% collected from Kafka, showing that when the file size is 4.4 gigabytes, the data processing speed in spark cluster mode is 2 minute 15 seconds, about 3 minutes 19 seconds faster than the local mode.

참고문헌 (Reference)

1 천승만, "제조로봇 고장예지진단을 위한 오픈소스기반 스마트 제조 빅데이터 플랫폼 구현" 대한임베디드공학회 14 (14): 187-195, 2019

2 류우석, "오픈소스 플랫폼 기반의 실시간 환자 대기시간 모니터링 시스템 설계" 한국정보통신학회 22 (22): 575-580, 2018

3 명노영, "아파치 스파크 활용 극대화를 위한 성능 최적화 기법" 한국정보처리학회 7 (7): 9-18, 2018

4 김태영, "실시간 자원 모니터링을 활용한 쿠버네티스 스케줄러 프레임워크 구현" 대한임베디드공학회 15 (15): 129-137, 2020

5 손형섭, "디지털 뉴딜 관련 데이터 법제의 입법 방향에 관한 연구" 한국헌법학회 27 (27): 203-252, 2021

6 한동희, "다중 웹사이트 관리를 위한 동작 기반 웹 크롤러 설정 구조 설계" 한국정보과학회 27 (27): 98-103, 2021

7 김학래, "국가 데이터의 의미적 표현과 연계를 위한 데이터맵 지식 모델" 한국디지털콘텐츠학회 22 (22): 491-499, 2021

8 방정미, "공공데이터 활용 실재화를 위한 데이터기반행정" 비교법문화연구소 21 (21): 87-126, 2021

9 "software"

10 "https://kafka.apache.org/27/documentation.html#connect"

11 "https://gitlab-gemscrc.gwnu.ac.kr/dgkim1108/bigdata-an alysis-platform"

12 "https://en.wikipedia.org/wiki/YAML"

13 "https://docs.docker.com/engine/install/ubuntu/"

14 "https://docs.docker.com/compose/"

15 "https://cloud.google.com/containers/?hl=ko"

16 "http://www.riss.kr/index.do"

17 S. M. Baek, "Spark Two Programming for the Bigdata Analysis" Wikibooks 1-630, 2018

18 장래영, "OpenFaaS 기반 AI 분석 서비스 시스템 구축" 한국콘텐츠학회 20 (20): 97-106, 2020

19 A. Eiermann, "On a Fog Computing Platform Built on ARM Architectures by Docker Container Technology" 2017 (2017): 71-86, 2017

20 C. W. Tien, "KubAnomaly : Anomaly Detection for the Docker Orchestration Platform with Neural Network Approaches" 1 (1): 1-20, 2019

21 P. Hoenisch, "Four-Fold Auto-Scaling on a Contemporary Deployment Platform Using Docker Containers" 2015 (2015): 316-323, 2015

22 고세윤, "Apache Spark를 활용한 대용량 데이터의 처리" 한국통계학회 29 (29): 1077-1094, 2016

23 I. S. Jeong, "Apache Kafka : From Application Development to Pipeline, Internet of Things data hub Construction" Hanbit Media 1-388, 2020

24 A. Hosny, "AlgoRun : a Docker-based Packaging System for Platform-agnostic Implemented Algorithms" 32 (32): 2396-2398, 2016

25 D. G. Kim, "A Study on Bigdata Collection and Processing ENgine of Docker Based" 593-594, 2021

26 H. R. Yu, "2020 Data Industrial White-paper : Market Status" (23) : 110-141, 2020