로봇을 위한 제스처 인식 시스템

김유진 대구대학교 2008 국내석사

인간은 일상생활에서 제스처와 같은 비 언어적 수단을 이용하여 많은 정보 를 전달한다. 따라서 인간과 정보 시스템이 자연스럽게 대화 할 수 있는 인터 페이스 구축을 위한 제스처 인식에 관한 연구는 필수적이다. 제스처를 인식한 다는 것은 인체 각 부위가 시간축에 대하여 어떤 형상의 변화를 가지는가를 알아내는 것을 의미한다. 그러나 제스처의 종류는 무한하고 인체는 매우 복잡 한 구조를 지닌 3차원 물체이므로 이를 자동으로 인식하는 것은 매우 어려운 일이다. 초기에는 인체 각 부위의 관절에 부착된 센서를 통해서 형상 변위값 을 입력하여 시공간적인 형상 패턴을 추출하고 제스처를 인식 하였다. 이 방 법은 장치를 몸에 붙이는 과정이 복잡하고 초기 교정이 어려우며 연결 케이블 로 인하여 자유로운 제스처의 입력이 불가능 하여 현재는 거의 사용되지 않고 있다. 별도의 입력 장비를 사용하지 않고 컴퓨터와 인간 간의 상호작용 기술로서 제스처 인식 방법은 많은 연구가 행해지고 있으며, 먼 거리나 잡음의 환경에 서 인간과 컴퓨터 간의 정보 전달 수단이 될 수 있으며 더 나아가 로봇 동작 제어, 사용자의 서비스 제공, 게임, 가상현실 등에 활용 될 수 있다. 카메라 영상 기반 제스처 인식 기술을 인간-로봇 상호작용 목적으로 활용하 기 위해서는 일반 범용 사용자를 대상으로 인식 대상 제스처의 종류를 10개 이내로 제한하여 복합 특징 정보 값 및 복합 제스처 인식기를 사용하여야 한 다. 또한 posture 정보를 같이 활용하면 보다 나은 인식 성능을 보장 할 수 있 을 것이다. 인간-로봇 간의 자연스러운 상호작용을 위한 제스처 인식 기술의 개발은 기계적이고 인위적인 컴퓨터 인터페이스 환경을 자연스러운 컴퓨팅 환 경으로 대체 할 수 있다는 데 그 중요성을 가지게 될 것이다. 본 논문에서는 애완용 강아지 로봇에 장착된 카메라로부터 보이는 사용자의 제스처를 인식하여 로봇에 명령을 하달하는 로봇제어 방법론을 제안한다. 제 안된 인터페이스를 이용하여 사용자는 로봇을 손 제스처만으로 제어할 수 있 다. 사용자의 손 제스처를 인식하기 위해 인터페이스는 실시간으로 입력되는 연속 영상으로부터 사용자의 손을 검출하고 손의 움직임, 모양 등의 특징들을 추출한다. 손을 검출하기 위해 HSI 색상공간에서 피부색을 정의하고, 연속 영 상에서 검출된 손 영역의 중심점 이동으로 손의 움직임 특징들을 추출한다. 손 모양은 의미 있는 제스처를 구분하고, 이를 알기위해 손 영역의 중심점을 중점으로 하는 원과 손 영역과의 교차점 수를 이용하여 찾는다. 추출된 특징 들로부터 양자화된 심볼열은 손 제스처 인식을 위해 은닉 마르코프 모델에 입 력된다. 본 논문에서 애완용 강아지 로봇을 제어하기 위한 명령으로 앉아, 일어서, 엎드려, 악수의 4가지 제스처를 정의하여 사용하였다. 실험 결과에서 제안한 인터페이스를 통해 사용자가 제스처를 사용하여 애완용 강아지 로봇을 제어 할 수 있음을 보인다. 사용된 로봇은 머리와 다리를 구성하는 서보모터 6개, 다리를 구성하는 서보모터 8개를 사용하였고, 이는 로봇 머리를 상하좌우 조 정이 가능하게 해주고, 최대한의 자연스러운 이동을 가능하게 해 준다. Human transmits using non language mean like gesture language. Therefore, research about gesture recognition for interface construction that it can communicate between human and information system are indispensability. Gesture recognition means find out that part of human body has some shape variation about time axis. But, kinds of gesture are infinity and because of human body is three dimension object having very complex structure, therefore, it is very difficult recognize human body automatically. Early time it inputs through attached sensor on region of human body articulation shape displacement value and it abstracts shape pattern of the hour space and recognized gesture. This method is almost useless at the present day because of it complicate process which attaches a system in the body and revision is difficult initially and free gesture input is impossible with the connection cable. It does not use input device especially, gesture recognition method as interaction technology between computer and human is many research comes to do, it can be between computer and human information delivery means in distant distance or environment of noise, further, it can be applied in robot action control, use service provision, game, virtual reality, and so on. Gesture recognition technique of camera image base which it applies in interaction purpose of human and robot that it must use a composition feature information value and the composition gesture recognizer who does the general purpose user in the object limits recognition object gesture type within 10 things. Also it can be guarantee more excellent recognition performance if use posture information together. Development of gesture recognition technique for natural interaction of between human and robot will have an importance which it can replace mechanical and artificially computer interface environment by natural computing environment. In this paper, suggest the robot control methodology that it recognizes user gesture seeing from the camera on pet dog robot others an other in the robot. User use suggested interface can control robot only hand gesture. Interface For recognition user hand gesture detects user hand from it be inputed real time continuous image and extract features of motion of hand, hand shape end so on. Defines skin color in HSI color space For detect hand, extracts features of hand motion with movement of hand center point detected in continuous image. Hand shape distinguishs meaning gesture, we detect for know that the number of intersection between the circle with center point of hand area and hand area. Quantified symbols from extracted features input in Hidden markov model for hand gesture recognition. In this paper, we defines four commands of sit down, stand up, lie flat, shake hands for controls pet robot. We are visible that user can control use gesture through interface suggested in experiment result. The robot used six servo-motor composed of head and leg, eight servo-motor composed of leg, it can control robot head the upper direction, down, the left, the right, it possible maximum natural movement.

Gestures as Diagnostics and Gestures as Tools : Two Relationships Between Gestures and Learning in Role Play

김동균 서울교육대학교 교육전문대학원 2014 국내석사

Gesture analysis has been conducted by many researchers (e.g. Beattie 2003; Goldin-Meadow 1998, 1999; McNeill 1985, 1992, 2004, 2005). The correlation between gesture and intonation has also been studied (e.g. Bolinger, 1983; Loehr, 2004) but almost all of these researches have been done in first language. This thesis investigates how gestures vary with second language developmental and learning level and how second language learners vary their intonation with gesture. Thus this thesis investigates two potentially useful relationships between gestures and learning for teachers and students. The study investigates and compares two very different groups of second language learners. The same role play was taught and videotaped in Korean for Australian students who were grade 6 and 7, and in English for Korean grade 3 to grade 6 students. Gestures were coded on the basis of McNeill (1992) and Beattie (2003): into pure action, beat, and indexical. The students’ voices were then analyzed with Praat software to check the relationship between gesture and intonation. The findings were twofold. Firstly, the Australian 6 graders who are just in the beginners’ level of learning the Korean language gestured far more than Korean pre-intermediate level equivalents who are in the same age and thus same developmental level. Secondly, students change their articulation and intonation as they internalized the target sentences. This could be seen through analyzing the voice files using praat in which students showed gesture change from meaningless to synchronized gesture. Based on these findings, the two following functions of gestures produced by language learning students were drawn. Firstly, gesture can be other-directed. Gesture enables teachers to spot the learning difficulties students have. Through students’ gestures, teacher can assess how students follow the lesson and their learning stages. Students use some gestures as communication strategy. They showed pure action gestures for help or expressing nervousness. Secondly, gesture can be self-directed. I could see students’ using gestures to help themselves learn language. They produced beat gestures to recall words and indexical synchronized gestures for intonation. Some pedagogical conclusions for different developmental levels can be drawn from these two findings and these two functions. Grade three and four students still enjoy role play. It appears to go well with their developmental level. They still like animated stories and role play. They feel less burden doing role play in front of whole class than grade five and six students. It seems plausible that if teachers give more elaborate instruction and demonstration with meaningful gestures before student’s role play, young learners may copy gestures which they wouldn’t normally produce in their developmental level. This will enhance them to learn language. For grade 5 and grade 6 students who don’t like role play anymore, we cannot assume that gesture in animated role play no longer has a learning function. One of the possible solutions for this would be using UCC for them to learn English in regards to this study. These mature students like real drama instead of simple animated role play. If we ask them to make UCC by groups using the expressions they learned in the classroom, then this will awaken their sleeping motivation. They will be eager to make roles and voluntarily participate in making UCC in real life. They will be also more involved in their characters and try to do their best to convey the meaning. In doing so, they will produce more gestures voluntarily due to their friends’ videotaping, which will lead them to enhance acquisition. Gesture is a tool to assess students and means to learn language through self-directed learning. As the students gesture, their speaking gets more energy. Students learn language through gesture. From this perspective, all teachers who teach young learners may wish to use more gestures and show good demonstrations in role play so that students acquire language better.

3D Human and Hand Skeletal Features with Continuous Hand Gesture Spotting and Classification

누엔 녹 황 전남대학교 2019 국내석사

Hand gestures are one of the most intuitive and natural ways for communication between human and computer. Recently, the role of hand gesture recognition has become more significant in human-computer interaction applications due to its convenience and naturalness. Hand gestures recognition based method is the topic that is increasingly attracted much research and development. In this paper, we present a novel approach for continuous dynamic hand gesture recognition. Our approach contains two main modules. Firstly, in the gesture spotting module, the video sequence with continuous gestures are pre-segmented into isolated gestures. Secondly, the gesture classification module classifies the segmented gestures. In the gesture spotting module, the motion of the hand palm and finger movements are fed into a Bidirectional Long Short-Term Memory (Bi-LSTM) network for gesture spotting purpose. In the gesture classification module, three residual 3D Convolution Neural Networks based on ResNet architectures (3D_ResNet) and one Long Short-Term Memory (LSTM) network are combined to efficiently utilize the combination of multiple data channels such as RGB, Optical Flow, Depth and 3D position of key joints. The promising performance of our approach is obtained by experiments conducted on three public datasets – Chalearn LAP ConGD dataset, 20BN-Jester, and NVIDIA Dynamic Hand gesture Dataset. Our approach achieves mean Jaccard Index of 0.6159, which outperforms the state-of-the-art methods on Chalearn LAP ConGD dataset.

(A) gesture classifier tree algorithm for motion gesture sensor

조영택 Graduate School, Korea University 2013 국내석사

Recently, portable handheld devices such as smart phones and tablets have become the hottest category in the electronics industry. With the recent advancements in distribution and usage of handheld devices, the range of services offered has broadened from the conventional mobile communication. With the increase in usage and complexity of handheld devices comes a need for a high user interface. The most commonly used user interface system for the execution of various complicated functions is the touchscreen. The touchscreen is the most accurate user interface system in use, presenting excellence both in simplicity and design. However, because of the versatility of handheld devices in its ability to be used in various environments, the touchscreen has shown some limitations in its use. For example, this system has constraints when wearing gloves or getting wet or dirt hands. To solve these problems, gesture sensors, which are similar in concept with the air mouse where no direct contact with the device is necessary, have been introduced. Currently, there are three types of gesture sensors depending on its input source. First is the motion-based system which uses a gyroscope and an accelerator sensor. Although the motion-based system can recognize a variety of different gestures, it has the disadvantage where the user must always be holding on to the device. Next is the vision-based system which uses a camera. This system has the advantage of being able to recognize various motions as well as 3D gestures, but the camera must be on during the whole time it is active, thus having high power consumption. In order reduce active power during use of gesture sensors, proximity-based system was introduced Proximity-based systems use IR LED and IR receivers to detect simple gestures and reduces power consumption own to one-tenth that of vision-based systems. Conventional proximity-based systems have the disadvantage of requiring a high form factor since a set distance between the IR LED and IR receiver is needed in order to detect gestures. In this thesis, in order to overcome the disadvantage of the high form factor found in conventional proximity-based gesture sensors, two IR receivers embedded on a single chip and an IR LED was used. The goal of this thesis is to use the proposed algorithm to solve the problem associated with bringing the two IR receivers close to each other and to implement a gesture sensor capable of recognizing three different gestures of right/left/click from a distance of 10cm and above. The proposed gesture sensor was implemented on a FPGA board using Verilog HDL. Evaluation results show that the system is able to recognize many gestures with 95% accuracy in real time.

Design of a proximity-based motion gesture sensor using a single LED

박호영 Graduate School, Korea University 2013 국내박사

With the fast growth in popularity of portable handheld devices the daily participation of these devices such as tablets, media players, e-readers, and smart phones is becoming tremendously expanded which results human-machine interfaces (HMIs) to operate under various circumstances. Recently, these devices operate with complex functions that require complicated HMIs. Current HMIs can be classified into four types: touch-based, motion-based, vision-based, and proximity-based systems. The touch-based system is one of the most common and natural methods either using fingers or a stylus. However, when the gloved, wet or dirty fingers touch, touching cannot be detected. The single-handed interaction is supported by motion-based system without touching or moving screen while interacting or by using an external controller. The vision-based system enables user to interface without touching the screen by using embedded camera and image processing. However it needs high power consumption and high computational cost. It is a huge drawback as a portable handheld device. The proximity-based motion gesture sensor (MGS) system was proposed to overcome this obstacle. For low power dissipation and contactless gesture recognition, the system uses a proximity sensor assembled by a photodiodes (PD) and two IR LEDs. According to the distance and the angle between an object and IR LEDs, the intensity of reflected IR lights varies. The simple gestures can be extracted with change of the intensity and gesture recognition algorithm. The system needs three separate placements for two IR LEDs and a proximity sensor, resulting in large form factor (FF) when defining FF as the boundary of sensing system. That can be a design limitation. In this thesis, we propose a novel proximity-based MGS system that assembled by a proximity sensor having two PDs and an off chip LED. The past method detects the difference of time between the received lights from two IR LEDs when an object moves one side from the other. However, for the minimum detection margin, the method needs a minimum distance between the two LEDs. The proposed optical block method divides the view angles of the two PDs for the light reflected by an object, and needs only a single LED. When the distance between a proximity sensor and an LED of the proposed system is 4mm, the distance between two LEDs of a conventional system is 40mm. By using proposed system, the form factor can be reduced to one tenth. Also, low noise sensor to amplify input signal from embedded small photodiodes (180μm by 180μm) is required. For low noise sensor, filtering technique is very important to reduce ambient light noise and electrical device noise. Generally, conventional filtering technique to reduce noise is band-pass-filter (BPF). But, BPF has disadvantage such as large area, low linearity and reliability. To overcome disadvantage of BPF synchronous sample/filtering method was proposed. This technique overcome disadvantage of BPF such as large area, low linearity and reliability, but it can’t filter DC noise and harmonic noise of modulation frequency. In this thesis, new synchronous sample/filtering method with Correlated Double Sampling (CDS) is proposed that compensate disadvantage of conventional method. The proposed method can filter DC noise and even harmonic noise of modulation frequency. Performance of sensor can be improved as using the proposed filtering method. The sensor chip was fabricated in CIS 0.18μm technology and the chip size is 1.2 mm by 1.7 mm. Test boards consist of sensor with embedded photodiodes, LED driver and FPGA board to detect gesture. MGS system was tested in various conditions such as optical block, distance between photodiode and object so on to verify motion gesture operation.

실시간 경로 변경을 위한 제스처 기반의 NUI 활용기법

유홍철 동국대학교 2014 국내석사

In this paper, we have proposed a heuristic approach to use gesture based on NUI technology. According to the hand shape we have defined five gestures. This method employs a flexible use of gestures to guarantee the most freedom for users. Even those who have never before used a computer can be accommodated by using gestures from life experience. We used 3D path modification as an application for our research target. We can use hand gesture to create path planning and edit it. Our core method relies on detecting 3D collision points. By detecting collision points, we can test whether a certain event could occur or not, and then we know the condition about the real time path. 3D collision point detection is based on intersection detection, and intersection in nature is a type of interaction. We did a simulation experiment about city path modification. Experiment resulted that computer can accurately identify a valid gesture. By using gesture it can effectively change the path. Among other applications, this solution can be used in virtual military maps and car navigation.

A Multi-touch Gesture Vocabulary Design Methodology for Mobile Devices

박원규 포항공과대학교 일반대학원 2012 국내박사

This study aims to propose a methodology for designing multi-touch gesture vocabularies applicable on mobile devices. The specific research objectives are 1) Proposing an integrated procedure for designing gesture vocabularies; 2) Proposing a systematic framework to analyze and create gesture vocabularies; 3) Validating the proposed methodology through a case study for designing mobile web browsing gestures; 4) Suggesting best multi-touch gesture vocabularies for mobile web browsers (MWBs). The integrated procedure involved both empirical approach and analytical approach. The empirical approach collected intuitive gestures from end-users for mobile web browsing commands. Users defined appropriate gestures intuitively themselves. On the other hand, the analytical approach created gestures based on a systematic framework. The framework consisted of gesture features and gesture elements, and the two attributes were combined to create various gestures. Designers generated gesture vocabularies analytically based on the framework. The analytical approach followed the empirical approach in the procedures. Basic gesture elements that constitute multi-touch gestures were identified. Extrinsic postures or motions were decomposed into the smallest units, and the categorized units became gesture elements. The gesture elements allowed observers to describe gestures concretely, and moreover the elements were used to analyze the usage patterns of gestures. Taxonomy of multi-touch gestures was also developed based on the basic elements. This study identified 6 basic elements of multi-touch gestures in mobile devices. The proposed elements were ‘posture’, ‘location’, ‘pose’, ‘touch’, ‘path’, and ‘device’. The usefulness and practicality of the methodology were validated through a case study. The case study identified mobile web browsing gestures to improve short-cut interactions. The initial gestures for the commands were collected from 36 end-users who did not have any experience with gesture interaction on web browsers. A total of 642 gestures were collected in the empirical approach. Six volunteers created gestures based on the systematic design framework, and they generated 314 gestures in the analytical approach. The proposed methodology helped designers to find appropriate gesture vocabularies for various commands on mobile devices. Gesture candidates were selected from the gestures which were collected by the empirical approach and created by the analytical approach. The selected candidates were evaluated by the 22 subjects. Their intuitiveness and subjective satisfaction scores are compared with each other to identify the most suitable gesture vocabularies. Finally, 11 best gestures and 38 top group gestures were identified for mobile web browsing commands. This study tried to find cognitively and physically appropriate web browsing gestures to end-users. However, technological issues such as confliction between different commands and restriction of recognition algorithm were not considered. Based on gestures identified in this study, further study considering these issues should be conducted in the future.

WiFi-Based Low-Complexity Gesture Recognition using Categorization

김지수 서울대학교 대학원 2022 국내석사

As smart homes and augmented reality (AR) become popular, the convenient human-computer interaction (HCI) methods are also attracting attention. Among them, many researchers have paid attention to gesture recognition that is simple and intuitive for humans. Camera-based and sensor-based gesture recognition have been very successful, but have limitations including privacy issues and inconvenience. On the other hand, WiFi-based gesture recognition using channel state information (CSI) does not have these limitations. However, since the WiFi signal is noisy, Deep learning (DL) models have been commonly utilized to improve the gesture recognition performance. DL models require large training data, large memory, and high computational complexity, resulting in long latencies that disrupt real-time systems. To solve this problem, support vector machines (SVMs) that require less computation and memory than powerful deep learning models can be utilized. However, the SVM shows poor performance when there are many target classes. In this paper, we propose a categorization method that can divide ten gestures into four categories. Since only two or three target gestures belong to each category, a traditional machine learning model like support vector machine (SVM) can achieve high accuracy while requiring less computation and memory consumption than the DL models. According to the experimental results, when using the SVM alone, the accuracy is about 58%. However, when used with categorization, it can improve up to 90%. Furthermore, the gesture recognition performance of the DL models can also be improved by combining the proposed categorization method if the hardware has sufficient memory and computational complexity. 스마트홈과 증강현실(AR)이 보편화되면서 편리한 인간-컴퓨터 상호작용 방식도 주목받고 있다. 그 중 많은 연구자들이 인간에게 간편하고 직관적인 Gesture Recognition에 주목해 왔다. 카메라 기반 및 센서 기반 Gesture Recognition은 매우 성공적이었지만 개인 정보 보호 문제 및 불편함 등의 한계가 있다. 반면, 채널 상태 정보(CSI)를 이용한 WiFi 기반 Gesture Recognition은 이러한 제한이 없다. 그러나 WiFi 신호에 노이즈가 많기 때문에 Gesture Recognition 성능을 향상시키기 위해 딥러닝 모델이 일반적으로 활용되었다. 딥러닝 모델은 대규모 훈련 데이터와 대용량 메모리가 필요하고 높은 계산 복잡도로 인해 실시간 시스템을 방해하는 긴 지연 시간을 초래한다. 이 문제를 해결하기 위해 강력한 딥러닝 모델보다 연산과 메모리가 덜 필요한 SVM을 활용할 수 있다. 하지만 SVM은 대상 클래스가 많을수록 성능이 크게 저하되는 문제가 있었다. 따라서 gesture를 여러 범주로 나눔으로써 대상 클래스를 줄이는 범주화 방법을 제안한다. gesture segment라고 하는 gesture unit을 찾는 것이 범주화 방법의 핵심이다. 각 Gesture는 고유한 gesture segment 개수를 가지므로 gesture를 숫자로 범주화할 수 있다. 예를 들어, 밀기 및 당기기와 같은 연속 gesture에는 두 개의 segment가 있다. 첫 번째 segment는 밀기이고 두 번째 segment는 당기기이다. 사람들이 현재 gesture segment를 중지하고 다음 gesture segment를 수행할 때 segment 사이에 short pause가 발생한다. 우리는 CSI 진폭의 변동을 분석하여 이러한 short pause를 찾을 수 있음을 관찰했다. CSI의 진폭은 사람이 움직일 때 더 커지고 그 반대도 마찬가지이다. 이를 바탕으로 진폭의 변화를 이용하여 short pause를 찾고 gesture segment를 나누는 범주화 방법을 제안한다. 범주화 이후 범주에 해당하는 SVM이 CSI 데이터를 사용하여 발생한 gesture를 결정한다. 제안된 범주화 방법은 98.5%의 정확도를 보였고 최종적으로 10개의 gesture에 대해 SVM의 성능을 약 30% 향상시킬 수 있었다. 또한 우리가 제안한 시스템은 딥러닝 모델을 활용한 비교대상에 비해 훨씬 적은 메모리와 지연 시간을 필요로 한다. 이 결과는 제안된 방법이 준수한 정확도를 가지며 AP 및 IoT 장치와 같은 제한된 하드웨어에도 배포할 수 있음을 보여준다.

Stereo camera-based hand gesture image recognition system using a thinning method

LI, XIANGHUA Sungkyunkwan University 2012 국내석사

In this paper, we propose a real-time hand gesture recognition system, using a thinning method that utilizes a stereo camera and implements a Korean chess game system. We implement a depth map in the hand detection portion, which uses the sum of absolute differences method, based on the acquired right and left image of detecting the foreground object. We use the YCbCr space, instead of the RGB space, to obtain the skin color region that can reduce the unnecessary hand image based on the acquired hand detection image. The next step uses the convex hull method to detect the boundary point and obtain the center coordinate of the hand and calculates the largest distance between the center point and the boundary point as the radius. This radius is used to make a rectangle, as in the ROI. Then, we use the distance value again to remove the background image in the ROI and detect the foreground image as the hand image. Finally, we use a blob labeling method to get the largest blob and remove the other small blobs that obtain the hand images without the noise caused by the computing distance from stereo matching. The hand gesture recognition system uses the Zhang and Suen thinning algorithm to obtain feature points, angles, and distances. This system recognizes five types of hand gestures. The proposed method achieves an average recognition rate of 83%. At last, we applied the gesture recognition result to a Korean chess game system that can move the chess pieces by the players hand.

무용표현 형식으로써의 Gesture에 관한 연구

김금광 이화여자대학교 대학원 1990 국내석사