스마트폰 다종 데이터를 활용한 딥러닝 기반의 사용자 동행 상태 인식

김길호,최상우,채문정,박희웅,이재홍,박종헌 한국지능정보시스템학회 2019 지능정보연구 Vol.25 No.1

As smartphones are getting widely used, human activity recognition (HAR) tasks for recognizing personal activities of smartphone users with multimodal data have been actively studied recently. The research area is expanding from the recognition of the simple body movement of an individual user to the recognition of low-level behavior and high-level behavior. However, HAR tasks for recognizing interaction behavior with other people, such as whether the user is accompanying or communicating with someone else, have gotten less attention so far. And previous research for recognizing interaction behavior has usually depended on audio, Bluetooth, and Wi-Fi sensors, which are vulnerable to privacy issues and require much time to collect enough data. Whereas physical sensors including accelerometer, magnetic field and gyroscope sensors are less vulnerable to privacy issues and can collect a large amount of data within a short time. In this paper, a method for detecting accompanying status based on deep learning model by only using multimodal physical sensor data, such as an accelerometer, magnetic field and gyroscope, was proposed. The accompanying status was defined as a redefinition of a part of the user interaction behavior, including whether the user is accompanying with an acquaintance at a close distance and the user is actively communicating with the acquaintance. A framework based on convolutional neural networks (CNN) and long short-term memory (LSTM) recurrent networks for classifying accompanying and conversation was proposed. First, a data preprocessing method which consists of time synchronization of multimodal data fromdifferent physical sensors, data normalization and sequence data generation was introduced. We applied the nearest interpolation to synchronize the time of collected data from different sensors. Normalization was performed for each x, y, z axis value of the sensor data, and the sequence data was generated according to the sliding window method. Then, the sequence data became the input for CNN, where feature maps representing local dependencies of the original sequence are extracted. The CNN consisted of 3 convolutional layers and did not have a pooling layer to maintain the temporal information of the sequence data. Next, LSTM recurrent networks received the feature maps, learned long-term dependencies from them and extracted features. The LSTM recurrent networks consisted of two layers, each with 128 cells. Finally, the extracted features were used for classification by softmax classifier. The loss function of the model was cross entropy function and the weights of the model were randomly initialized on a normal distribution with an average of 0 and a standard deviation of 0.1. The model was trained using adaptive moment estimation (ADAM) optimization algorithm and the mini batch size was set to 128. We applied dropout to input values of the LSTM recurrent networks to prevent overfitting. The initial learning rate was set to 0.001, and it decreased exponentially by 0.99 at the end of each epoch training. An Android smartphone application was developed and released to collect data. We collected smartphone data for a total of 18 subjects. Using the data, the model classified accompanying and conversation by 98.74% and 98.83% accuracy each. Both the F1 score and accuracy of the model were higher than the F1 score and accuracy of the majority vote classifier, support vector machine, and deep recurrent neural network. In the future research, we will focus on more rigorous multimodal sensor data synchronization methods that minimize the time stamp differences. In addition, we will further study transfer learning method that enables transfer of trained models tailored to the training data to the evaluation data that follows a different distribution. It is expected that a model capable of exhibiting robust recognition performance against changes in data that is not considered in the model learning stage wil... 스마트폰이 널리 보급되고 현대인들의 생활 속에 깊이 자리 잡으면서, 스마트폰에서 수집된 다종 데이터를바탕으로 사용자 개인의 행동을 인식하고자 하는 연구가 활발히 진행되고 있다. 그러나 타인과의 상호작용 행동 인식에 대한 연구는 아직까지 상대적으로 미진하였다. 기존 상호작용 행동 인식 연구에서는 오디오, 블루투스, 와이파이 등의 데이터를 사용하였으나, 이들은 사용자 사생활 침해 가능성이 높으며 단시간 내에 충분한 양의 데이터를 수집하기 어렵다는 한계가 있다. 반면 가속도, 자기장, 자이로스코프 등의 물리 센서의 경우 사생활 침해 가능성이 낮으며 단시간 내에 충분한 양의 데이터를 수집할 수 있다. 본 연구에서는 이러한 점에 주목하여, 스마트폰 상의 다종 물리 센서 데이터만을 활용, 딥러닝 모델에 기반을 둔 사용자의 동행 상태 인식 방법론을 제안한다. 사용자의 동행 여부 및 대화 여부를 분류하는 동행 상태 분류 모델은 컨볼루션 신경망과 장단기기억 순환 신경망이 혼합된 구조를 지닌다. 먼저 스마트폰의 다종 물리 센서에서 수집한 데이터에 존재하는 타임 스태프의 차이를 상쇄하고, 정규화를 수행하여 시간에 따른 시퀀스 데이터 형태로 변환함으로써 동행 상태분류 모델의 입력 데이터를 생성한다. 이는 컨볼루션 신경망에 입력되며, 데이터의 시간적 국부 의존성이 반영된 요인 지도를 출력한다. 장단기 기억 순환 신경망은 요인 지도를 입력받아 시간에 따른 순차적 연관 관계를학습하며, 동행 상태 분류를 위한 요인을 추출하고 소프트맥스 분류기에서 이에 기반한 최종적인 분류를 수행한다. 자체 제작한 스마트폰 애플리케이션을 배포하여 실험 데이터를 수집하였으며, 이를 활용하여 제안한 방법론을 평가하였다. 최적의 파라미터를 설정하여 동행 상태 분류 모델을 학습하고 평가한 결과, 동행 여부와 대화 여부를 각각 98.74%, 98.83%의 높은 정확도로 분류하였다.

스마트폰 기반 산소 결핍 모니터링 장치

노병국 ( Byoung Gook Loh ) 한국안전학회(구 한국산업안전학회) 2015 한국안전학회지 Vol.30 No.6

O2-deficiency related accidents occur every year and the most effective way to prevent them is to measure O2 concentration in air with a properly-calibrated O2 monitoring device before entering low-O2 areas. An electro-chemical sensor, Texas Instrument gas platform, and iPhone are used to construct a smartphone-based O2 monitoring device. The smartphone based O2 measuring approach offers advantages of small size, accessibility, internet-connectivity, and programmability in comparison to conventional O2 measuring devices. Multiple gas sensors can be conveniently interfaced to single smartphone, allowing for creating a network of gas sensors distributed across workplaces and remote monitoring via existing mobile communication network. To check proper function of the O2 monitoring device the sensor was exposed to shallow and deep human breaths. The readings decreased immediately after being exposed to exhalation and recovered during inhalation to a calibrated level of 20.9%. When readings decreased below a preset warning value of 19.5%, a low O2 warning was successfully activated on the smartphone.

Smartphone-based O₂ Deficiency Monitoring Device

노병국,Loh, Byoung Gook The Korean Society of Safety 2015 한국안전학회지 Vol.30 No.6

$O_2$-deficiency related accidents occur every year and the most effective way to prevent them is to measure $O_2$ concentration in air with a properly-calibrated $O_2$ monitoring device before entering low-$O_2$ areas. An electro-chemical sensor, Texas Instrument gas platform, and iPhone are used to construct a smartphone-based $O_2$ monitoring device. The smartphone based $O_2$ measuring approach offers advantages of small size, accessibility, internet-connectivity, and programmability in comparison to conventional $O_2$ measuring devices. Multiple gas sensors can be conveniently interfaced to single smartphone, allowing for creating a network of gas sensors distributed across workplaces and remote monitoring via existing mobile communication network. To check proper function of the $O_2$ monitoring device the sensor was exposed to shallow and deep human breaths. The readings decreased immediately after being exposed to exhalation and recovered during inhalation to a calibrated level of 20.9%. When readings decreased below a preset warning value of 19.5%, a low $O_2$ warning was successfully activated on the smartphone.

Implementation of Rule-based Smartphone Motion Detection Systems

Eon-Ju Lee(이언주),Seung-Hui Ryou(유승희),So-Yun Lee(이소윤),Sung-Yoon Jeon(전성윤),Eun-Hwa Park(박은화),Jung-Ha Hwang(황정하 ),Doo-Hyun Choi(최두현) 한국컴퓨터정보학회 2021 韓國컴퓨터情報學會論文誌 Vol.26 No.7

스마트폰에 내장된 각종 센서를 통해 획득할 수 있는 정보는 사용자의 움직임, 상황 등을 파악하고 분석하는데 유용하게 활용될 수 있다. 본 논문에서는 스마트폰의 가속도 센서와 자이로스코프 센서에서 얻은 정보를 분석하여 ‘I’, ‘S’, ‘Z’ 모션을 인식하는 두 가지 규칙기반 시스템을 제안한다. 먼저, 각 모션에 대한 가속도 및 각속도의 특성을 분석한다. 이를 기반으로 두 가지 종류의 규칙기반 모션 인식 시스템을 제안하고 이를 안드로이드 앱으로 구현하여 각 모션에 대한 성능을 비교한다. 두 가지 규칙기반시스템은 각 모션에 대해서 90% 이상의 인식률을 보이며 앙상블을 이용한 규칙기반 시스템은 다른 시스템보다 향상된 성능을 보인다. Information obtained through various sensors embedded in a smartphone can be used to identify and analyze user’s movements and situations. In this paper, we propose two rule-based motion detection systems that can detect three alphabet motions, ‘I’, ‘S’, and ‘Z’ by analyzing data obtained by the acceleration and gyroscope sensors in a smartphone. First of all, the characteristics of acceleration and angular velocity for each motion are analyzed. Based on the analysis, two rule-based systems are proposed and implemented as an android application and it is used to verify the detection performance for each motion. Two rule-based systems show high recognition rate over 90% for each motion and the rule-based system using ensemble shows better performance than another one.

Smartphone-based colorimetric detection for inverse opal photonic hydrogel sensor

박하빈,이원목 한국공업화학회 2019 한국공업화학회 연구논문 초록집 Vol.2019 No.0

Here we report a smartphone-based colorimetric detection of inverse opal photonic hydrogel sensor which causes the shift of diffraction wavelength following reversible changes in physical dimensions. Identifying color changes is basically an eye, but it may have tendency of leading to uncertain results. On the other hand, the spectrophotometer is commonly used as precision analysis equipment in traditional colorimetric detection, but it is bulky, expensive and time-consuming. Thus, a smartphone-based mobile sensing approaches can be a great alternative over conventional methods by leveraging its built-in functions. The advantages of various analyte-sensitive and reversible hydrogel-based colorimetric sensor and the portable smartphone enable real-time monitoring platforms for many applications including medical diagnosis.

이동 로봇 제어를 위한 안드로이드 스마트폰 어플리케이션 개발

김민지,배설봉,주문갑,이원창 한국정보기술학회 2014 한국정보기술학회논문지 Vol. No.

In this paper, we propose an Android smartphone application to control the posture of mobile robots. The Android smartphone application transmits various commands of posture to a mobile robot by Bluetooth communication. The posture commands are perceived from user motion by collecting data values from acceleration sensor and orientation sensor in the Android smartphone. Pseudo-augmented reality is aimed, where the user is controlling and observing the mobile robots through a camera screen displaying the robot information together, to prevent dispersion of the user's sight. Experiment using a mobile car and a mobile ship shows 91.53% recognition rate of the posture command by the posture of Android smartphone. 본 논문에서는 이동 로봇의 자세를 제어하기 위한 안드로이드 스마트폰 어플리케이션을 개발하였다. 안드로이드 스마트폰 어플리케이션은 이동 로봇의 컨트롤러 역할을 하며, 블루투스 통신을 통해 이동 로봇에게 자세 명령을 전달한다. 자세 명령은 안드로이드 스마트폰의 방향 센서와 가속도 센서의 데이터 값의 변화에 따라 인식되어 이동 로봇에 전달된다. 제안된 방법에서는 증강 현실 기법을 사용하여 사용자가 스마트폰의 정보가 같이 표시되는 카메라 화면을 통하여 이동 로봇을 주시하고 제어하므로 실제 운용시 사용자의 시선 분산을 막는다. 안드로이드 스마트폰의 자세에 따라 자세 명령을 인식하는 실험결과 이동 로봇은 평균 91.53%의 인식률을 보였다.

스마트폰의 CMOS 영상센서를 이용한 광용적맥파 측정방법 개발

김호철(Kim, Ho-Cheal),정원식(Jeong, Won-Sik),이권희(Lee, Kwon-Hee),남기창(Nam, Ki Chang) 한국산학기술학회 2015 한국산학기술학회논문지 Vol.16 No.6

맥파는 심전도와 같이 자율신경계를 통해 생리적 반응을 측정하는 신호이지만, 손가락에 센서 하나만 부착시키면 되기 때문에 상대적으로 신호의 측정이 간편하다는 장점을 가지고 있어 u-Healthcare 분야에서의 활용이 용이하다. 따라서 본 연구의 목적은 스마트폰 카메라의 CMOS 영상 센서를 활용하여 맥파를 비침습적으로 측정하는 방법 중의 하나인 광용적 맥파를 획득하고 이로부터 스트레스 여부를 판단하는 휴대형 시스템을 개발하여 u-Healthcare 분야에서의 활용 가능성을 확 인하는 것이다. 이를 위해 광용적맥파를 별도의 센서에 의한 측정이 아닌 스마트폰 카메라에서 획득되는 영상 데이터를 활용 하여 광용적맥파를 획득한 후 분석하였다. 또한 확보된 광용적맥파 영상신호 데이터를 이용하여 심박변이도와 스트레스 지 수를 별도의 호스트 장비 없이 스마트폰만을 이용해 사용자에게 제공 하였다. 또한 부가적으로 스마트폰에 부착가능한 별도 의 하드웨어 디바이스를 개발함으로써 획득된 데이터의 신뢰도 및 정확성을 향상시켰다. 실험결과를 통해 스마트폰의 카메 라 영상을 활용하여 광용적맥파 신호를 통한 심박수 측정과 스트레스의 정도를 분석하기 위한 스트레스 지수 추출이 가능함 을 확인할 수 있었다. 본 연구에서는 상용화된 제품 또는 정형화된 센서가 아닌 스마트폰의 카메라를 이용하기 때문에 상용 화된 외부 센서에 의한 광용적맥파 신호 보다는 해상도가 떨어지는 단점이 있음에도 불구하고 결과 데이터의 신뢰도 향상을 위한 별도의 추가외부 장치 개발 및 여러 가지 최적화 알고리즘을 통해 신뢰성 있는 데이터를 확보할 수 있어 u-Healthcare 장비로써의 활용 가능성을 확인할 수 있었다. Pulse wave is the physiological responses through the autonomic nervous system such as ECG. It is relatively convenient because it can measure the signal just by applying a sensor on a finger. So, it can be usefully employed in the field of U-Healthcare. The objects of this study are acquiring the PPG (Photoplethysmography) one of the way of measuring the pulse waves in non-invasive way using the CMOS image sensor on a smartphone camera, developing the portable system judging stressful or not, and confirming the applicability in the field of u-Healthcare. PPG was acquired by using image data from smartphone camera without separate sensors and analyzed. Also, with that image signal data, HRV (Heart Rate Variability) and stress index were offered users by just using smartphone without separate host equipment. In addition, the reliability and accuracy of acquired data were improved by developing additional hardware device. From these experiments, we can confirm that measuring heart rate through the PPG, and the stress index for analysis the stress degree using the image of a smartphone camera are possible. In this study, we used a smartphone camera, not commercialized product or standardized sensor, so it has low resolution than those of using commercialized external sensor. However, despite this disadvantage, it can be usefully employed as the u-Healthcare device because it can obtain the promising data by developing additional external device for improvement reliability of result and optimization algorithm.

Creating Covert Channel by Harnessing Shapley Values from Smartphone Sensor Data

Jun-Won Ho 한국인터넷방송통신학회 2021 Journal of Advanced Smart Convergence Vol.10 No.3

In this paper, we devise a Shapley-value-based covert channel in smartphones. More specifically, unlike ordinary use of Shapley value in cooperative game, we make use of a series of Shapley values, which are computed from sensor data collected from smartphones, in order to create a covert channel between encoding smartphone and decoding smartphone. To the best of our knowledge, we are the first to contrive covert channel based on Shapley values. We evaluate the encoding process of our proposed covert channel through simulation and present our evaluation results.

충돌 및 균형상실 사고 방지를 위한 스마트폰 연동 전동 휠체어

김태현,제양규,이종선 한국기계기술학회 2019 한국기계기술학회지 Vol.21 No.5

This research has been conducted to develop a prevention module of collision and balance lost in electric wheelchairs. Ultrasonic sensors were used for detecting obstacles. Giro and acceleration sensors were used for measuring the tilt angle of wheel chair frame. Kalman filters were employed for filtering high-frequency noise of acceleration sensor and removing zero drift of the Giro sensor. The presence and location of obstacles were detected, and their information was sent to the user smartphone. When the tilt angle of the wheelchair frame exceeded a certain value, a warning signal was set to operate. Raspberry Pi camera module was used to transfer rear view image data to the smartphone to help the driver acknowledge obstacles in the rear side. A DSP processor with high computing performance was necessary to realize a real-time processing of the sensor and image data. Our research showed that a prevention module of collision and balance lost connected to a smartphone can be realized at a considerable low price.

가속도 센서와 방향 센서를 이용한 패스워드 추측 공격에 대응하는 보안 키패드

김익수,최종명 한국지식정보기술학회 2014 한국지식정보기술학회 논문지 Vol.9 No.4

According to recent studies, data generated from smartphone’s built-in accelerometer and gyroscope sensors can be used to infer users’ passwords. Currently, the secure keypad which is being used in smartphone apps is vulnerable to password attacks using sensor data. In this paper, we propose a secure keypad against password guessing attacks with accelerometer and gyroscope sensors. In the keypad, the rows of keys on the proposed keypad is randomly changed whenever a user inputs a letter of the password. Accordingly, it is very difficult to find out the user’ password using accelerometer and gyroscope sensor data. Moreover, the proposed keypad uses fake key buttons to further reduce the success rate of the attack. Users do not have to memorize fake keys. Users only need to touch the fake keys on the proposed keypad when they input their passwords. Because attackers cannot distinguish fake keys from touched keys, they cannot find out users’ passwords with accelerometer and gyroscope sensors. Therefore users can safely use a variety of internet services.