앙상블 경험적 모드 분해를 이용한 수질자료의 이상치 탐색

박상수,박노석,김성수,조귀래,윤석민 대한환경공학회 2021 대한환경공학회지 Vol.43 No.3

Objectives:This study was conducted to propose a new methodology for efficiently identifying and removing various outliers that occur in data collected through automated water quality monitoring systems. In the present study, water temperature data were collected from domestic G_water supply system, and the performance of the proposed methodology was tested for water temperature data collected from domestic G_water supply system. Methods:We applied the following analytical procedure to identify outliers in the water quality data: First, a normality test was performed on the collected data. If normality condition was satisfied, the Z-score was used. However, if the normality condition was not satisfied, outliers were identified using the quartile, and the limitations of the existing methodology were analyzed. Second, we decomposed the intrinsic mode function using empirical mode decomposition and ensemble empirical mode decomposition for the collected data, and then considered the occurrence of modal mixing. Finally, a group of intrinsic mode functions was selected using statistical characteristics to identify outliers. In addition, the performance of the method was verified after removing and interpolating outliers using regression analysis and Cook’s distance. Results and Discussion:In the case of water temperature data, as normality condition was not satisfied, outlier identification was carried out by applying the modified quartile method. It was confirmed that outliers distributed within the seasonal component could not be identified at all. In the case of empirical mode decomposition, modal mixing occurred because of the effect of outliers. However, in the case of the ensemble empirical mode decomposition, modal mixing was resolved and the distinct seasonal components were decomposed as intrinsic mode functions. The intrinsic mode functions were synthesized, which showed statistical correlation with the raw water temperature data. As a result of developing a regression model using the synthesized intrinsic mode functions and raw water temperature data and performing outlier search based on Cook’s distances, we concluded that various outliers distributed within the seasonal component could be effectively identified. Conclusions:Considering that satisfactory results could be derived from statistical analysis of the data collected from the automated water quality monitoring system, it can be concluded that outlier identification procedures are essential. However, in the case of the conventional univariate outlier search method, it is apparent that the outlier search performance is significantly poor for data with strong inherent variability, and the interpolation method for the searched outlier cannot be performed. Conversely, the outlier identification method based on ensemble empirical mode decomposition and regression analysis proposed in this study shows excellent discrimination performance for outliers distributed in data with strong inherent variability. Moreover, this method has the advantage of reducing the analyst’s dependence on subjective judgment by presenting statistical cutoff criteria. An additional advantage of the method is that data can be interpolated after removing outliers using intrinsic mode functions. Therefore, the outlier search and interpolation method proposed in this study is expected to have greater applicability as a more effective analysis tool compared to the existing univariate outlier search method. 목적:본 연구는 국내 상수도 자동수질측정망을 통해 수집되는 자료에서 발생 가능한 다양한 이상치들을 효율적으로 탐색 및 제거 위한 방법론을 제안하기 위해 수행되었다. 이를 위해 국내 G_정수장으로부터 수온자료를 수집하였으며, 수집된 자료를 대상으로 이상치 방법론에 따른 적용 효과를 검정하였다. 방법:본 연구에서 수질자료의 이상치 탐색을 위해 적용한 분석 절차는 다음과 같다. 첫째, 수집된 수온자료에 대해 정규성 검정을 수행하고 정규성을 만족하는 경우 Z-score, 정규성을 만족하지 않는 경우 사분위수를 활용하여 이상치를 탐색하고 기존 방법론의 한계점에 대해 분석한다. 둘째, 수온자료에 대해 경험적 모드 분해 및 앙상블 경험적 모드 분해를 활용하여 고유진동함수들을 분해한 후 모드 믹싱에 발생에 대해 고찰한다. 최종적으로 고유진동함수들의 통계적 특성치를 활용해 이상치를 식별할 기준 고유진동함수 집단을 선별한 후 회귀분석과 Cook 통계량의 절사 기준을 활용해 이상치를 제거 및 보간 후 그 성능을 검증한다. 결과 및 토의:수온자료의 경우 정규성을 만족하지 못하며, 수정 사분위 방법을 적용하여 이상치 탐색을 수행한 결과 계절 성분 내에 분포하는 이상치들은 전혀 식별할 수 없다는 결과를 확인하였다. 경험적 모드 분해의 경우 이상치들의 효과로 인해 모드 믹싱 현상이 발생하였으나, 앙상블 경험적 모드 분해에서는 모드 믹싱이 해결되어 뚜렷한 계절 성분이 고유진동함수로서 분해되는 것으로 나타났다. 그리고 앙상블 모드 분해로부터 구해진 고유진동함수 중 원시 수온자료와 통계적 관계성이 높은 신호들을 합성하였다. 합성된 고유진동함수와 원시 수온자료를 활용해 회귀 모형을 개발하고, Cook 통계량 근간으로 이상치 탐색을 수행한 결과 계절 성분 내에 분포하는 다양한 이상치들을 효과적으로 탐색할 수 있는 것으로 분석되었다. 결론:상수도 자동수질측정망을 통해 수집되는 자료들로부터 합리적인 통계분석 결과를 도출하기 위한 과정에서 이상치 탐색 작업은 필수적이라고 할 수 있다. 하지만 기존의 단변량 이상치 탐색 기법의 경우 고유 변동성이 강하게 분포하는 자료에 대해 이상치 탐색 성능이 현저히 떨어지며, 탐색된 이상치에 대한 내삽 방안도 제시하지 못한다는 한계가 명확하다. 반면, 본 연구에서 제시한 앙상블 경험적 모드 분해 및 회귀분석 기반의 이상치 탐색 방법은 고유 변동성이 강한 자료 내에 분포하는 이상치들에 대한 식별 성능이 뛰어나며, 통계적 절사 기준을 제시함에 따라 분석자의 주관적 판단을 최소화 할 수 있는 장점이 있다. 또한 앙상블 경험적 모드 분해 분석으로부터 구해진 고유진동함수들을 이용해 이상치 제거 후 자료 보간이 가능하다는 장점이 있다. 따라서 기존의 단변량 이상치 탐색 기법의 적용성에 대한 한계를 고려할 때 본 연구에서 제시한 이상치 탐색 및 보간 방안은 보다 효과적인 분석 도구로서 적용 가능할 것으로 기대된다.

Temporal and spatial outlier detection in wireless sensor networks

Hoc Thai Nguyen,Nguyen Huu Thai 한국전자통신연구원 2019 ETRI Journal Vol.41 No.4

Outlier detection techniques play an important role in enhancing the reliability of data communication in wireless sensor networks (WSNs). Considering the importance of outlier detection in WSNs, many outlier detection techniques have been proposed. Unfortunately, most of these techniques still have some potential limitations, that is, (a) high rate of false positives, (b) high time complexity, and (c) failure to detect outliers online. Moreover, these approaches mainly focus on either temporal outliers or spatial outliers. Therefore, this paper aims to introduce novel algorithms that successfully detect both temporal outliers and spatial outliers. Our contributions are twofold: (i) modifying the Hampel Identifier (HI) algorithm to achieve high accuracy identification rate in temporal outlier detection, (ii) combining the Gaussian process (GP) model and graph‐based outlier detection technique to improve the performance of the algorithm in spatial outlier detection. The results demonstrate that our techniques outperform the state‐of‐the‐art methods in terms of accuracy and work well with various data types.

은행산업의 효율성 측정시 outliers[異狀値] 적출방법

박노경,김종호,이뢰 한국산업경제학회 2008 산업경제연구 Vol.21 No.1

본 논문은 Wilson(1993)의 방법을 이용하여 은행효율성을 측정하는 경우 발생 할 수 있는 異狀(outlier) DMU(decision making unit)를 적출하는 방법을 보여주었다. 즉, 3개년(2003년, 2004년, 2005년)의 국내 14개 은행들을 대상으로 생산기능접근법, 중개기능접근법, 부가가치기능접근법을 이용하여 실증분석을 시행하고 해석하였다. 실증분석의 주요한 결과는 다음과 같다. 첫째, 이상 DMU로 적출된 국민은행, 제일은행, 부산은행, 대구은행은 투입지향형 CCR 모형 하에서 세 가지 접근법 모두에서 효율적이지 않았으며, BCC모형에서는 모두 효율적이었다. 둘째, 부가가치접근법에서는 CCR, BCC 모형에서 모두 비효율적이었다. 셋째, 이상 DMU들을 효율성 수치 측면에서는 살펴보면 국민은행>제일 및 외환은행>부산 및 대구은행의 순서였다. 본 연구의 정책적인 함의는 다음과 같다. 한국은 행이나 금융감독원의 은행분야 정책입안가 들은 본 분석에서 사용한 이상 DMU를 적출하는 방법을 이용하여 매우 특이한 투입-산출구조를 갖는 은행들에 대해서는 기타 비효율적인 보통의 은행들과 다른 그룹을 형성토록 하여 경쟁을 유도하는 정책적인 방안을 마련해야만 한다. The purpose of this paper is to introduce the new method when measuring bank efficiency developed by Wilson(1993) for detecting outliers in DEA analysis by using the 3 years(2003, 2004, and 2005) panel data under production function, intermediation function, and value-added function approaches. The main empirical results of this paper are as follows. First, empirical results show that Kookmin Bank, First Bank, Busan Bank, and Deagu Bank are outliers. All these banks are not efficiency in terms of input-oriented CCR model under production function, intermediation function, and value-added function approaches. But BCC model shows efficiency. Under value-added function approach, CCR and BCC model did not show the efficient score for outliers. Second, the efficient ranking orders of outliers are Kookmin Bank, First Bank and Foreign Exchange Bank, Busan Bank with Daegu Bank. The policy implications to the Korean banks and planners are that the method which this paper used to find out outliers should be recommended for grouping the efficient banks, inefficient banks and outliers with others and also for introducing the competitive policy among the outliers and efficient banks. Another policy implication is that for introducing suitable method for the each bank, the policy planner should check out the input and output compositions of efficient banks or outliers after finding out outliers by using the Wilson (1993) method which this paper introduced.

Statistical Method for Real-Time Detection of Travel Time Outliers

Moataz Bellah Ben Khedher(모타즈 케들러),Duk Geun Yun(윤덕근),Jung Gon Sung(성정곤) 한국산학기술학회 2021 한국산학기술학회논문지 Vol.22 No.10

도로 이용자나 도로관리자에게 교통 분석이나 경로 선택에 있어 통행시간은 중요한 정보이다. 그러나 수집된 개별 통행시간에 이상치(outlier)가 존재함으로 인해서 수집된 통행시간 정보는 항상 그 값을 대표한다고 보기에 한계가 존재한다. 통행시간 추정을 위한 기존의 방법은 방대한 과거 데이터를 사용하여 계수 추정(parametric models)을 위한 다양한 모형을 사용하였다. 기존 전통적인 방법에서 Outlier를 검지하고, 처리하여 신뢰성 있는 통행시간을 추정하기 위해 본 연구에서는 Non-parametric outlier filtering 방법인 Outskewer 방법을 제안하였다. Outskewer 적용시 기존에 사용하던 Sliding Window 방법 대신 분석할 데이터양이 충분하지 못한 구간에서는 알고리즘이 충분히 분석할 수 있는 데이터양을 확보할 수 있도록 직전의 데이터를 더 추가할 수 있는 Extendable Moving Window 기법을 적용하였다. 제안된 모형을 평가하기 위하여 다차로 도로에서 수집된 DSRC(Dedicated Short Range Communication)를 사용하였다. 모형 평가시 통행시간의 참 값(ground truth)의 부재로 인하여 그래프와 통계의 유의성 검증을 통하여 모형을 평가하였다. 통행시간에 대한 통계적 검증결과 최대 상대 오차(maximum relative error)는 20%보다 적게 나타났고, Correct Classification Rate는 90% 이상으로 나타났다. 결과를 고려할 때 제안된 방법론은 통행시간 추정 시 Outlier를 처리하는데 있어서 기존의 방법론보다 더 신뢰성 있게 나타나 도로이용자와 관리자에게 더 정확한 통행시간을 제공할 수 있을 것으로 기대하였다. Travel time information plays a crucial role in supporting road users and transportation agencies with precise data that can be used for traffic analysis and route choosing. However, the acquired data does not always show a representative value of the travel time due to the presence of outliers. The outlier must hence be removed, and various techniques of outlier filtering rely on parametric models that require extensive historical data for parametric estimation. In this study, we propose a non-parametric outlier filtering method for a reliable travel time estimation. As part of the study, an Outskewer method was adopted and enhanced to overcome its deficiency and to be able to detect outliers in online mode. Dedicated short-range communication probe data, collected from a multi-lane highway, was used for the quality assessment. In lack of ground truth records, the performance of the proposed algorithm was evaluated qualitatively, based on graphs and quantitatively using the confidence interval method. Visual inspection of graphical results shows the satisfactory performance of the present method. The hypothesis test for travel time data quality indicates a maximum relative error of less than 20% and a percentage correct classification of higher than 90%. Hence, the proposed methodology can serve as an alternative to other parametric methods used for travel time outlier treatments and can support the delivery of more accurate information for road travelers and transportation practitioners.

Outlier Detection Method for Time Synchronization

Lee, Young Kyu,Yang, Sung-hoon,Lee, Ho Seong,Lee, Jong Koo,Lee, Joon Hyo,Hwang, Sang-wook 항법시스템학회 2020 Journal of Positioning, Navigation, and Timing Vol.9 No.4

In order to synchronize a remote system time to the reference time like Coordinated Universal Time (UTC), it is required to compare the time difference between the two clocks. The time comparison data may have some outliers and the time synchronization performance can be significantly degraded if the outliers are not removed. Therefore, it is required to employ an effective outlier detection algorithm for keeping high accurate system time. In this paper, an outlier detection method is presented for the time difference data of GNSS time transfer receivers. The time difference data between the system time and the GNSS usually have slopes because the remote system clock is under free running until synchronized to the reference clock time. For investigating the outlier detection performance of the proposed algorithm, simulations are performed by using the time difference data of a GNSS time transfer receiver corrected to a free running Cesium clock with intentionally inserted outliers. From the simulation, it is investigated that the proposed algorithm can effectively detect the inserted outliers while conventional methods such as modified Z-score and adjusted boxplot cannot. Furthermore, it is also observed that the synchronization performance can be degraded to more than 15% with 20 outliers compared to that of original data without outliers.

A Study of the Rainfall in Korea Using Outlier Detections

Ho Geun Son,Jae Joon Lee 위기관리 이론과 실천 2019 Crisisonomy Vol.15 No.10

연최대 강우자료의 이상치는 확률강우량을 산정하는 빈도해석에서 중요한 역할을 한다. 신뢰성 있는 빈도해석을 위하여 본 연구에서는 연 최대 강우자료의 통계적 검토에 의한 이상치 검정과 아울러 이상치 강우들을 선별하여 강우 발생 원인을 검토하는 정성적 분석을 수행한다. 이를 위하여 국내 57개 기상관측지점의 24시간 강우자료들을 대상으로 Grubbs and Beck 방법, IACWD 방법, 수정 z-score 방법과 Box-plot 방법을 이용하여 이상치 검정을 실시하였다. 선정된 고이상치의 강우사상에 대해 발생 원인을 장마, 태풍, 집중호우와 전선형 강우로 분류하여 분석하였다. 분석결과, 방법들의 비교를 통해 Grubbs and Beck 방법이 다른 방법에 비하여 가장 적게 이상치가 발생하였다. 강우 발생 원인 분석에서 태풍 51.0%, 집중호우 35.7%, 전선형 강우 7.2%와 장마 6.1% 순서로 나타났으며, 내륙 지점에서는 집중호우, 해안과 인접한 지역들은 태풍에 의한 이상치 들이 많이 발생하였다. 검정을 통해 발생된 이상치를 기각하여 산정한 확률강우량과 이상치 검정전의 확률강우량을 비교하였다. The outliers in annual maximum rainfall can play an important role in the frequency analysis that estimates the probability rainfall. To ensure reliability of the frequency analysis, this study detects the outliers and conducts a qualitative analysis to investigate the causes of rainfall based on selected outliers. The outliers in annual maximum 24-hr rainfalls at 57 stations in Korea were analyzed by four different methods, such as Grubbs and Beck method, IACWD method, modified z-score method and box-plot method. The causes of the selected rainfall events as high outliers were classified into changma, typhoon, severe rain storm and frontal rainfall. The comparison of the methods shows that less outliers were found with Grubbs and Beck method than the other methods. It is found that 51.0% of the outliers occurred due to typhoon, followed by severe rain storm (35.7%), frontal storm (7.2%) and changma (6.1%). Rainfall events as outliers were mostly caused by severe rain storm in inland regions while they were caused by typhoon in coastal regions. The probability rainfall calculated without outliers from the test was compared with that before the test.

선형회귀모형에서의 이상치 탐색방법들의 비교 연구

박대인,강현철,한상태,최호식 한국자료분석학회 2013 Journal of the Korean Data Analysis Society Vol.15 No.1

This paper is concerned with the outlier detection methods in regression model. Various influence measures for detecting outliers are illustrated and compared via real data sets. Including single outlier detection method and three multiple outliers detection methods are considered : procedure based on the least median of squares estimation, the robust distance with the minimum volume ellipsoid estimator, and She, Owen (2011) procedure. Comparison studies are conducted using two data sets which are known to contain multiple outliers. It appears, in general, that all of these procedures are effective in identifying the outliers. However, procedures based on the least median of squares estimation and the robust distance with the MVE estimator are difficult to identification of outliers when masking or swamping effects exist in the data. On the other hand, the procedure proposed by She, Owen (2011) is effective to detect such outliers. 통계적 자료 분석에서 이상치 검출은 기본적인 분석과정으로 모델 선택, 추론, 평가 등 거의 모든 분석과정에 상당한 영향을 끼친다. 경우에 따라서는 식별된 이상치 자체가 중요한 정보를 담기도 한다. 본 논문에서는 분석모형으로 선형회귀모형을 고려하고 이에 대한 여러 가지 이상치 탐색 방법들을 살펴보고자 한다. 이상치 탐색방법은 크게 지렛점 등의 전통적인 통계량 등을 활용하여 순차적으로 이상치를 식별하는 단일(single) 이상치 식별방법과 강건한 추정을 기반으로 여러 개의 이상치를 동시에 검출할 수 있는 다중(multilple) 이상치 식별방법(least median of square estimation, robust distance with minimum volume ellipsoid estimator)으로 구분할 수 있다. 본 논문에서는 다중 이상치를 포함하고 있다고 알려진 실제 자료를 통해, 최근 제안된 She, Owen (2011)의 축소추정법을 통한 다중 이상치 검출방법과 기존의 여러 방법들의 특징을 살펴보았다. 특히 수렁효과나 가면효과를 가진 자료에 대해서 She, Owen(2011)의 방법이 기존의 방법들보다 이러한 이상치들 간의 관계를 효과적으로 탐색할 수 있음을 확인하였다.

임계치 모형과 인공신경망 모형을 이용한 실시간 저수지 수위자료의 이상치 탐지

김마가,최진용,방재홍,이재주 한국농공학회 2019 한국농공학회논문집 Vol.61 No.1

Reservoir water level data identify the current water storage of the reservoir, and they are utilized as primary data for management and research ofagricultural water. For the reservoir storage management, Korea Rural Community Corporation (KRC) installed water level stations at around 1,600agricultural reservoirs and has been collecting the water level data every 10 minutes. However, various kinds of outliers due to noise and erroneousproblems are frequently appearing because of environmental and physical causes. Therefore, it is necessary to detect outlier and improve the qualityof reservoir water level data to utilize the water level data in purpose. This study was conducted to detect and classify outlier and normal data usingtwo different models including the threshold model and the artificial neural network (ANN) model. The results were compared to evaluate theperformance of the models. The threshold model identifies the outlier by setting the upper/lower bound of water level data and variation data and bysetting bandwidth of water level data as a threshold of regarding erroneous water level. The ANN model was trained with prepared training datasetas normal data (T) and outlier (F), and the ANN model operated for identifying the outlier. The models are evaluated with reference data which werecollected reservoir water level data in daily by KRC. The outlier detection performance of the threshold model was better than the ANN model, butANN model showed better detection performance for not classifying normal data as outlier.

The Identification Of Multiple Outliers

Park, Jin-Pyo The Korean Data and Information Science Society 2000 한국데이터정보과학회지 Vol.11 No.2

The classical method for regression analysis is the least squares method. However, if the data contain significant outliers, the least squares estimator can be broken down by outliers. To remedy this problem, the robust methods are important complement to the least squares method. Robust methods down weighs or completely ignore the outliers. This is not always best because the outliers can contain some very important information about the population. If they can be detected, the outliers can be further inspected and appropriate action can be taken based on the results. In this paper, I propose a sequential outlier test to identify outliers. It is based on the nonrobust estimate and the robust estimate of scatter of a robust regression residuals and is applied in forward procedure, removing the most extreme data at each step, until the test fails to detect outliers. Unlike other forward procedures, the present one is unaffected by swamping or masking effects because the statistics is based on the robust regression residuals. I show the asymptotic distribution of the test statistics and apply the test to several real data and simulated data for the test to be shown to perform fairly well.

Outlier detection and variable selection via difference based regression model and penalized regression

InHae Choi,Chun Gun Park,Kyeong Eun Lee 한국데이터정보과학회 2018 한국데이터정보과학회지 Vol.29 No.3

This paper studies an efficient procedure for the outlier detection and variable selection problem in linear regression. The effect of outliers is added in linear regression as a mean shift parameter, nonzero or zero constant. To t this mean shift model, most penalized regressions have used some adaptive penalties on the parameters to shrink most of the parameters to zero. Such penalized models do select the true variables well, but do not detect the outliers correctly. To overcome this problem, we first determine a group of possibly suspected outliers using difference-based regression model (DBRM) and add the group to the linear model as the parameters of the effect of each suspected outlier. Then, we perform outlier detection and variable selection simultaneously using Lasso regression or Elastic net regression for the linear regression with the effect term of each suspected outlier added. The proposed method is more efficient than the previous penalized regression. We compare the proposed procedure with other methods using a simulation study and apply this procedure to the real data.