Towards intelligent child behavior recognition in indoor settings

다국어 초록 (Multilingual Abstract)

Recognizing child behavior in indoor environments is critical for ensuring safety and well-being. By nature, children are exploratory and often involved in various activities which lead to injury. Similarly, they are sometimes involved in activities that damage the prosperity of the house. Besides the hazardous interaction, they also exhibit aggressive behaviors in response to frustration, stress, and the inability to self-regulate emotions. In the long term, such behavior leads to disrupted social skills, hinders emotional regulation, and overall prevents the healthy mindset development. However, traditional methods fall short in recognizing hazardous and aggressive behaviors in children due to the lack of available datasets. The lack of specialized datasets hinders the development of deep learning-based models for intelligent child behavior recognition in indoor environments. These result in unreliability, subjective reporting, and delayed response, despite the advancement of AI-based monitoring systems. In response, in this study, we contribute to intelligent child behavior recognition and present two major contributions to advance the research on child well-being. In the first part of this study, we introduce the Real-world Hazardous Activity Recognition dataset (RHAD). RHAD is a video dataset collected to capture various child hazardous activities in real-world indoor environments. Additionally, we propose a specialized model called HazardNet, designed to accurately recognize hazardous activities. Additionally, we perform comprehensive benchmarking of the SOTA methods. Our experimental results highlight that HazardNet exhibits better performance, around 9.2% higher in terms of accuracy, compared to the latest method, VideoMamba. The second part of this study presents the Child Aggressive Behavior Recognition Dataset (CABAD). A comprehensive video dataset for aggressive behavior recognition in indoor environments. Utilizing CABAD, we propose a model called CABA_Net, specifically designed to recognize aggressive behavior in children. Our results show that CABA_Net achieves better performance compared to CNN and transformer-based models. Through these contributions, we aim to contribute to intelligent child behavior recognition. The insights and analysis disseminated in these contributions provide a strong baseline for future research. These create a pathway to have more safer environment for children and also contribute to enhancing child well-being.

국문 초록 (Abstract)

컴퓨터 비전과 인공지능 기술의 발전은 인간 활동을 자동으로 분석하는 능력을 크게 향상시켜 왔습니다. 그러나 실내 환경에서의 아동 행동 이해는 여전히 어려운 연구 분야로 남아 있습니다. 이는 아동의 행동이 자발적이고 예측 불가능하며, 때로는 안전과 직결되는 특성을 가지기 때문입니다. 가정이나 교육 공간에서 아동은 신체적 위험, 정서적 불안, 혹은 생활 환경의 손상을 초래할 수 있는 행동을 자주 보입니다. 그 중요성에도 불구하고 기존 연구는 주로 성인 또는 일반적인 아동 활동에 집중하고 있어, 위험하거나 공격적인 아동 행동을 충분히 반영하지 못하고 있습니다. 이로 인해 현재의 행동 인식 모델들은 조기 위험 감지나 실시간 안전 모니터링에 한계를 보입니다.

이러한 문제를 해결하기 위해 본 학위논문은 두 가지 주요 기여를 제안합니다. 첫 번째는 실제 실내 환경에서 발생하는 아동의 위험 및 생활공간 훼손 행동 12개 범주를 포함한 대규모 비디오 데이터셋인 RHAD(Real-world Hazardous Activity Dataset) 입니다. RHAD는 다양한 위험 수준, 환경적 변이, 객체 상호작용을 포함하여 현실적이고 복합적인 상황을 반영합니다. 본 연구는 이를 기반으로 효율적인 공간 특징 표현과 시간적 패턴을 통합한 하이브리드 모델 HazardNet을 제안하였습니다. 실험 결과, HazardNet은 여러 평가지표에서 기존 모델보다 우수한 성능을 보여 아동 안전 모니터링에 적합함을 입증하였습니다.

두 번째 기여는 아동에게서 흔히 관찰되는 6가지 공격 행동을 포함한 CABAD(Child Aggressive Behavior Activity Dataset) 입니다. 이를 활용하여 본 연구는 충동적이고 정서적으로 유발되는 행동의 특성을 효과적으로 포착할 수 있도록 설계된 시간-어텐션 기반 모델 CABA\_Net을 제안하였습니다. CABA\_Net은 다양한 촬영 조건에서도 높은 인식 성능을 보였으며, 아동 중심 모델 설계의 필요성을 확인시켰습니다.

종합적으로, 본 연구에서 제안한 데이터셋과 모델은 아동 행동 분석 분야의 발전에 기여하며, 향후 자동화된 위험 감지 시스템, 조기 개입 프로그램, 안전 중심 스마트 실내 환경 구축에 중요한 기반이 될 것입니다. 또한 실제 환경의 복잡성과 아동 행동의 특성을 반영함으로써, 아동의 안전과 전반적 삶의 질 향상을 위한 지능형 모니터링 기술 발전에 기여합니다.

목차 (Table of Contents)

• Chapter1. Introduction 1
• 1.1.Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
• A. Real-worldHazardousActivityDataset(RHAD) . . . . . . . . . . . . . . . . . . 2
• B. ChildAggressionBehaviorAnalysisDataset(CABAD) . . . . . . . . . . . . . . . 3
• 1.2. ProblemStatement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
• 1.3. Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
• 1.4. ScopeoftheStudy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
• Chapter2. RelatedWorks 9
• 2.1. ExistingDatasets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
• 2.2. ExistingMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
• Chapter3.Real-WorldHazardousActivityDataset 13
• 3.1. OurDataset:RHAD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
• A. DataCollection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
• B. AnnotationProcess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
• C. RHAD'sQuantitativeAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
• C.1. FrameDistribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
• C.2. ContextualUnderstanding. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
• D. EthicalConsiderations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
• 3.2. Proposed:HazardNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
• A. SpatialFeatureExtractorModule . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
• B. Frame-wiseTokenRegularization . . . . . . . . . . . . . . . . . . . . . . . . . . 23
• C. HybridTemporalRepresentationModule. . . . . . . . . . . . . . . . . . . . . . . 24
• D. FocalAdaptiveLearning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
• 3.3. PseudocodefortheProposedHazardNet . . . . . . . . . . . . . . . . . . . . . . . . 26
• 3.4. ExperimentalResults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
• A. Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
• B. BaselineModels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
• C. DetectionResults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
• D. HazardousActivityRecognition . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
• D.1.GeneralizabilityAcrossRiskTypes . . . . . . . . . . . . . . . . . . . . . . . . 33
• D.2. Real-worldRobustnessAssessment. . . . . . . . . . . . . . . . . . . . . . . . 34
• E. AblationStudy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
• E.1. AblationStudyonCoreModulesofHazardNet . . . . . . . . . . . . . . . . . . 36
• E.2. ImpactofBatchSizeandOptimizer . . . . . . . . . . . . . . . . . . . . . . . . 37
• Chapter4.ChildAggressionBehaviorAnalysisDataset(CABAD) 38
• 4.1. Child-AggressiveBehaviorAnalysisDatasetDescription . . . . . . . . . . . . . . . 40
• A. DataCollection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
• B. DataAnnotationandPre-processing . . . . . . . . . . . . . . . . . . . . . . . . . 42
• C. DatasetQuantitativeAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
• D. EthicalConsiderations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
• E. ProposedModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
• F. FeatureExtractionwithMobileViT . . . . . . . . . . . . . . . . . . . . . . . . . . 45
• G. Short-TermTemporalModelingwithTCN. . . . . . . . . . . . . . . . . . . . . . 46
• H. Long-TermTemporalModelingwithAttentionLSTM. . . . . . . . . . . . . . . . 47
• I. ExperimentalSetupandResults . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
• I.1. Implementationdetails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
• I.2. EvaluationMetrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
• J. ComputationalCompelexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
• K. ComparitiveAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
• L. Visualresults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
• L.1.ModelAccuracyandLoss . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
• L.2. PrecisionandClassDifferentiationEvaluation . . . . . . . . . . . . . . . . . . . 54
• L.3. VisualAnalysisofPredictedandGroundTruthLabels . . . . . . . . . . . . . . . 55
• M. AblationStudy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
• M.1. EvaluationofFeatureExtractionModels. . . . . . . . . . . . . . . . . . . . . . 57
• M.2. ImpactofBatchSizeandOptimizers. . . . . . . . . . . . . . . . . . . . . . . . 58
• M.3. ImpactofLearningRateandDataAugmentation . . . . . . . . . . . . . . . . . 59
• M.4. ImpactofModelComponentsonPerformance . . . . . . . . . . . . . . . . . . 60
• M.5. PerformanceunderAdverseCondition . . . . . . . . . . . . . . . . . . . . . . . 61
• Chapter5. Conclusion 64
• 5.1. Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
• 5.2. FutureDirections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65