NER(중첩 개체 인식)기법을 적용하는 효과적인 로그 템플릿 추출 기법 = Effective Log Template Extraction Technique via Nested Named Entity Recognition

다국어 초록 (Multilingual Abstract)

Modern software systems generate massive log messages that serve as
crucial data for monitoring. Effective log analysis requires log parsing
to transform unstructured logs into structured representations. However,
existing token-based parsing methods often fail to recognize nested
variable structures, such as JSON objects or arrays, leading to parsing
errors where templates become either overly specific or general.

This study proposes an improved log parsing method utilizing Nested Named
Entity Recognition (NNER) to address these limitations. The proposed
approach identifies hierarchical structure within logs and generates
multiple candidate templates with various levels of generality. It then
applies the Minimum Description Length (MDL) principle to select the
optimal template set that balances model simplicity and data encoding
efficiency. Additionally, an iterative filtering strategy is implemented
to ensure time efficiency when processing large-scale datasets.

Empirical evaluations on real-world logs from Hadoop, Spark, and
Apache-IoTDB demonstrate that the proposed method improves parsing
accuracy by 0.098 to 0.230 compared to existing parsers. Notably, it
shows superior performance in logs with complex nested structures where
traditional methods fail. Furthermore, the results indicate that higher
parsing quality enhances the performance of anomaly detection tasks.
These findings highlight that advanced log structure analysis is a
critical prerequisite for robust system anomaly detection.

목차 (Table of Contents)

• 1. 개요 1
• 2. 배경 지식 2
• 1) 로그 2
• 2) 로그 파싱 2
• 3) 중첩된 개체명 인식 3
• 4) MDL(Minimum Description Length) 4
• 3. 관련 연구 6
• 1) 규칙 기반 로그 파싱 6
• 2) NER을 적용한 로그 파싱 11
• 3) LLM을 활용한 로그 파싱 11
• 4) 중첩된 개체명 인식 모델 11
• 4. 문제 정의 14
• 5. 로그 파싱 기법의 설계 17
• 1) 알고리즘 개요 17
• 2) 후보 템플릿 생성 18
• 3) 후보 템플릿 집합 생성 20
• 4) 그룹화 22
• 5) 템플릿 간의 포함 관계를 활용한 후보 템플릿 집합 생성 23
• 6) 필터링 25
• 6. 평가 27
• 1) 실험 세팅 27
• 2) 로그 데이터 세트 27
• 3) 정답 로그 집합 29
• 4) 템플릿 정확도 비교 30
• 5) MDL 비용 비교 35
• 6) 이상탐지 성능 비교 39
• 7) 시간 성능 비교 40
• 7. 결론 44