(A) Rule-based Data Grouping Method for Personalized Log Analysis System in Big Data Computing

다국어 초록 (Multilingual Abstract)

In this paper, we proposed a rule-based data grouping method and personalized log analysis system using the method. And we performs a performance evaluation for verifying performance improvement. For the performance improvement, we studied R, statistical analysis tool, and MongoDB, one of Not Only SQL (NoSQL) databases. Then, we analyzed problems and limitations of current big data analysis system.
Nowadays, big data is being more important issue. Furthermore, big data trend is moving towards to provide personalized services to customers. And a recent report showed 73% of collected log data were used to perform the big data analysis. Hence, the log data is important source in big data analysis. Big data analysis is based on complex statistical analysis methods. It need to analyze various data type include unstructured data and semi-structured data for providing personalized services to customer. And big data can be valuable information when it is analyzed and derived ‘Big Question’ and ‘Big Answer’. Especially, log data is analyzed to find valuable informations. For the big data analysis, professional statistical tools such as R, SAS are needed. We used R that have a lot of packages consist of various analysis methods. R is based on single processing and in-memory processing because data size is small in the past. However, nowadays, big data analysis need high cost and resources when it processed in memory.
Therefore, big data analysis system have to consider distributed and parallel processing technologies to process in high performance. In the case of parallel processing, we can consider various parallel processing packages developed by many developers. And we can apply distributed database such as MongoDB and Cassandra or big data management system like Hadoop for the distributed processing. MongoDB, one of commonly used NoSQL, is a document store. it have features such as schema-free and scalable. However, general analysis or MapReduce using MongoDB shows a low performance and has a limitation such as performing limited analysis. Also, Hadoop, big data management system, be able to handle huge data by using MapReduce technology. But Hadoop is hard to optimize a workflow and analyze data by using various advanced analysis method.
In this paper, to solve these problems and limitations, we proposed Personalized Log Data Analysis System (PLAS) with the rule-based log data grouping method. By using the proposed method, grouping rule is created. And collected log data is stored by the grouping rule. The rule-based log data grouping method decrease overhead of pre-processing and analysis. Also, automated analysis system can be constructed by using the method because when the system cannot find matched rule with collected log data, it create grouping rule automatically.
Therefore, by using the proposed PLAS and method, data scientists perform various analysis methods and a performance can be improved. The PLAS provides automated grouping and supports personalized analysis. At that time, it be able to perform workflow optimization and distributed processing based on MongoDB and R. Also, mainly functions of the PLAS are constructed based on R interface to provide a familiar environment with data scientists. And we evaluated a performance of analysis by using the rule-based log data grouping method and it showed a performance is improved about 57 ~ 88 % because the PLAS provides shortest distance of query path.

목차 (Table of Contents)

• Abstract i
• List of Contents iii
• List of Figure iv
• List of Table v
• Chapter 1. Introduction 1
• Chapter 2. Related work 3
• 2.1. NoSQL 3
• 2.2. MongoDB 4
• 2.3. Statistical Analysis Tool ‘R’ 5
• 2.4. Data Grouping 6
• Chapter 3. Personalized Log data Analysis System 7
• 3.1. System Architecture 8
• 3.2. Performance Influencing Factors of the PLAS 11
• 3.3. Rule-based Log Data Grouping Method 12
• 3.4. Summary of Contributions 16
• Chapter 4. Numerical Analysis 17
• 4.1. Evaluation Environment 17
• 4.2. Total Processing Time 17
• 4.2. Functional Analysis 18
• Chapter 5. Conclusion and Future works 22
• Reference 24