(An) integrative approach to pedestrian detection based on sparse optical flow and convolutional neural network model

다국어 초록 (Multilingual Abstract)

Visual information plays a very important role in interacting with the real world. In the field of computer science, image processing algorithms inspired by human visual information systems have been widely studied. Traditional research topics include optical flow algorithms, which extract a motion vector between images using brightness constancy assumptions, and Histogram of Gradient (HOG) algorithms that detect objects by learning the boundary values of pixels. State-of-the-art, deep learning techniques in artificial intelligence research mimic neural structures in the human brain, and are an area of active development. However, small computers such as Micro Controller Unit (MCU) and Internet of Things (IOT) devices in industry create computational time restrictions to utilizing the state-of-the-art techniques.

In this dissertation, a light-weight object detection algorithm is designed to perform efficiently, even in situations where computer performance is limited. It reduces the computational load and increase the object detection performance by combining the existing traditional techniques and the latest techniques. Rather than using the entire state-of-the-art deep learning model, optical flow vectors are calculated to estimate moving objects in the beginning of the detection process. Then, in the last phase of the algorithm, machine learning and neural network models including deep learning are applied to extract image features for object classification.

The proposed method begins by introducing proposed feature point reset function, moving window and target estimator function based on the Lucas-Kanade method for sparse optical flow estimation. These functions improve performance when tracking moving objects and reduce computational time greatly. Next, it classifies the moving objects in the area extracted from the proposed optical-flow-based algorithm. When applying learning-based classification models, training data is a very important factor for the performance of the learning models. Training data relationships and confirmed coverage effects are determined among datasets by performing experiments to evaluate test datasets. In the end, a HOG+SVM machine learning model, You-Only-Look-Once version 5 (YOLOv5), and simple CNN-based Small Person Detection (SPD) deep learning models are used to classify the moving objects with the optical-flow-based algorithm. A learning-based classification model is applied to the object-existing Region of Interest (ROI) which is extracted from the optical-flow-based model. The optimal parameters are found by adjusting the crop size and resize ratio parameters according to the size of the ROI. Proposed CNN-based deep learning model SPD is operated on the proposed separated ROI method which is pre-process step. ROI is separated according to threshold of aspect ratio from the experiment analysis. Object detection performance such as FPS, recall, and precision are compared for the machine learning technique the HOG+SVM, HOG Multi-scale, OF+HOG+SVM, YOLOv5n, OF+YOLOv5n models, and the proposed deep learning technique OF+SPD model. The comparison results shows better performance in computational speed and object detection performance for proposed detection algorithm.

This work presents a pedestrian detection algorithm having high computational speed through sparse optical flow and a lightweight CNN-based neural network model. First, it employs an object localization function with a moving window detector, which mitigates the flaws of the sparse optical flow. Next, an SPD neural network model consisting of fewer CNN layers performs object classification. A performance comparison evaluation experiment of the proposed method was conducted upon a CPU device, which showed good detection performance and faster computation speed compared to existing object detection algorithms. This indicates that the proposed pedestrian detection algorithm could be applied using only a CPU device without the need for an expensive GPU device. The proposed algorithm can serve as an economic and efficient algorithm for real-time pedestrian detection in industrial areas where low-performance computing devices need to be used.

목차 (Table of Contents)

• 1 Introduction 1
• 1.1 Why efficient pedestrian detection?................... 1
• 1.2 Motivation and objective ........................ 2
• 1.3 Organization of dissertation....................... 3
• 2 Background 5
• 2.1 Moving object tracking......................... 6
• 2.1.1 Optical flow-based model.................... 6
• 2.1.2 Background subtraction problem................ 8
• 2.1.3 Application of moving object tracking . . . . . . . . . . . . . 10
• 2.2 Data for object detection ........................ 11
• 2.2.1 Datasource........................... 11
• 2.2.2 Dataprocessing......................... 14
• 2.3 Models for object detection....................... 17
• 2.3.1 Machine learning-based model................. 17
• 2.3.2 Deep neural network-based model............... 23
• 2.3.3 Application of learning-based models . . . . . . . . . . . . . 27
• 2.4 Summary of Chapter 2 ......................... 30
• 3 Moving object tracking based on sparse optical flow 31
• 3.1 Methods................................. 32
• 3.1.1 Corner feature reset....................... 33
• 3.1.2 Moving window detector.................... 34
• 3.1.3 Memorized estimator...................... 36
• 3.2 Experimental environment ....................... 37
• 3.2.1 Dataset ............................. 37
• 3.2.2 Missing box of corner parameters ............... 38
• 3.2.3 Evaluation of corner parameters ................ 41
• 3.3 Experimental results .......................... 44
• 3.3.1 Results with changes in window size. . . . . . . . . . . . . . 44
• 3.3.2 Results with changes in distance thresholds . . . . . . . . . . 46
• 3.3.3 Results with changes in estimation time . . . . . . . . . . . . 47
• 3.4 Comparisons .............................. 50
• 3.4.1 Comparison with other conventional methods . . . . . . . . . 50
• 3.4.2 Comparison in noisy videos .................. 52
• 3.5 Summary of Chapter 3 ......................... 55
• 4 Data selection for deep learning model 57
• 4.1 Methods................................. 58
• 4.1.1 Various portion of dataset ................... 59
• 4.1.2 Combination of dataset..................... 60
• 4.1.3 Relationship of dataset..................... 62
• 4.1.4 Coverage effects of dataset................... 63
• 4.2 Experimental environment ....................... 64
• 4.2.1 Camera measurement system.................. 64
• 4.2.2 Dataset ............................. 66
• 4.2.3 Deep learning model ...................... 67
• 4.3 Experimental results .......................... 69
• 4.3.1 Results with various portion of single dataset . . . . . . . . . 69
• 4.3.2 Results of combination of base dataset. . . . . . . . . . . . . 73
• 4.3.3 Results of relationship on custom dataset . . . . . . . . . . . 78
• 4.3.4 Results of coverage effects in custom dataset . . . . . . . . . 84
• 4.4 Summary of Chapter 4 ......................... 88
• 5 Pedestrian detection based on convolutional neural network model 89
• 5.1 Methods................................. 90
• 5.1.1 Small-sized pedestrian detection (SPD) network . . . . . . . 92
• 5.1.2 Region selection for moving objects . . . . . . . . . . . . . . 93
• 5.1.3 Pedestrian detection models .................. 95
• 5.1.4 Region partitioning....................... 96
• 5.2 Experimental environment ....................... 97
• 5.2.1 Dataset ............................. 97
• 5.2.2 HOG-based approach and deep learning models . . . . . . . . 98
• 5.3 Experimental results .......................... 98
• 5.3.1 Deep learning model ...................... 99
• 5.3.2 SPD network model ...................... 101
• 5.3.3 Comparison of detection performance . . . . . . . . . . . . . 107
• 5.4 Summary of Chapter 5 ......................... 111
• 6 Conclusions 113
• 6.1 Moving object tracking based on sparse optical flow . . . . . . . . . . 114
• 6.2 Data selection for deep learning model................. 114
• 6.3 Pedestrian detection based on convolutional neural network model . . 116
• 6.4 Future works .............................. 117
• Bibliography 119