효과적인 인터랙티브 비디오 저작을 위한 얼굴영역 기반의 어노테이션 방법

윤의녕(Ui Nyoung Yoon),가명현(Myeong Hyeon Ga),조근식(Geun-Sik Jo) 한국지능정보시스템학회 2015 지능정보연구 Vol.21 No.1

Many TV viewers use mainly portal sites in order to retrieve information related to broadcast while watching TV. However retrieving information that people wanted needs a lot of time to retrieve the information because current internet presents too much information which is not required. Consequentially, this process can’t satisfy users who want to consume information immediately. Interactive video is being actively investigated to solve this problem. An interactive video provides clickable objects, areas or hotspots to interact with users. When users click object on the interactive video, they can see additional information, related to video, instantly. The following shows the three basic procedures to make an interactive video using interactive video authoring tool: (1) Create an augmented object; (2) Set an object’s area and time to be displayed on the video; (3) Set an interactive action which is related to pages or hyperlink; However users who use existing authoring tools such as Popcorn Maker and Zentrick spend a lot of time in step (2). If users use wireWAX then they can save sufficient time to set object’s location and time to be displayed because wireWAX uses vision based annotation method. But they need to wait for time to detect and track object. Therefore, it is required to reduce the process time in step (2) using benefits of manual annotation method and vision-based annotation method effectively. This paper proposes a novel annotation method allows annotator to easily annotate based on face area. For proposing new annotation method, this paper presents two steps: pre-processing step and annotation step. The pre-processing is necessary because system detects shots for users who want to find contents of video easily. Pre-processing step is as follow: 1) Extract shots using color histogram based shot boundary detection method from frames of video; 2) Make shot clusters using similarities of shots and aligns as shot sequences; and 3) Detect and track faces from all shots of shot sequence metadata and save into the shot sequence metadata with each shot. After pre-processing, user can annotates object as follow: 1) Annotator selects a shot sequence, and then selects keyframe of shot in the shot sequence; 2) Annotator annotates objects on the relative position of the actor’s face on the selected keyframe. Then same objects will be annotated automatically until the end of shot sequence which has detected face area; and 3) User assigns additional information to the annotated object. In addition, this paper designs the feedback model in order to compensate the defects which are wrong aligned shots, wrong detected faces problem and inaccurate location problem might occur after object annotation. Furthermore, users can use interpolation method to interpolate position of objects which is deleted by feedback. After feedback user can save annotated object data to the interactive object metadata. Finally, this paper shows interactive video authoring system implemented for verifying performance of proposed annotation method which uses presented models. In the experiment presents analysis of object annotation time, and user evaluation. First, result of object annotation average time shows our proposed tool is 2 times faster than existing authoring tools for object annotation. Sometimes, annotation time of proposed tool took longer than existing authoring tools, because wrong shots are detected in the pre-processing. The usefulness and convenience of the system were measured through the user evaluation which was aimed at users who have experienced in interactive video authoring system. Recruited 19 experts evaluates of 11 questions which is out of CSUQ(Computer System Usability Questionnaire). CSUQ is designed by IBM for evaluating system. Through the user evaluation, showed that proposed tool is useful for authoring interactive video than about 10% of the other interactive video authoring systems.

K-Means 클러스터링을 적용한 향상된 CS-RANSAC 알고리즘

고승현 ( Seunghyun Ko ),윤의녕 ( Ui-nyoung Yoon ),( Jumabek Alikhanov ),조근식 ( Geun-sik Jo ) 한국정보처리학회 2017 정보처리학회논문지. 소프트웨어 및 데이터 공학 Vol.6 No.6

이미지를 기반으로 하는 증강현실 시스템에서 가상의 객체를 실제 영상에 저작할 때 생기는 이질감을 줄이기 위해서는 실제 영상에 저작된 가상객체의 방향과 위치에 대해 정확하게 추정을 해야 하며, 이때 호모그래피를 사용한다. 호모그래피를 추정하기 위해서는 SURF와 같은 특징점을 추출하고 추출된 특징점들을 통해 호모그래피 행렬을 추정한다. 호모그래피 행렬의 추정을 위해서 RANSAC 알고리즘이 주로 사용되고 있으며, 특히 RANSAC에 제약 조건 만족 문제(Constraint Satisfaction Problem)와 여기에 사용되는 제약조건을 동적으로 적용하여 속도와 정확도를 높인 DCS-RANSAC 알고리즘이 연구되었다. DCS-RANSAC 알고리즘에서 사용된 이미지 그룹 데이터는 수동적인 방법을 통해 직관적으로 분류되어 있지만 특징점 분포 패턴이 다양하지 않고, 이미지들을 정확하게 분류하기가 어려워서 이로 인해 알고리즘의 성능이 저하되는 경우가 있다. 따라서 본 논문에서는 K-means 클러스터링을 적용하여 이미지들을 자동으로 분류하고 각 이미지 그룹마다 각기 다른 제약조건을 적용하는 KCS-RANSAC 알고리즘을 제안한다. 제안하는 알고리즘은 머신러닝 기법인 K-means 클러스터링을 사용하여 전처리 단계에서 이미지를 특징점 분포 패턴에 따라 자동으로 분류하고, 분류된 이미지에 제약조건을 적용하여 알고리즘의 속도와 정확도를 향상시켰다. 실험결과 본 논문에서 제안하는 KCS-RANSAC이 DCS-RANSAC 알고리즘에 비해 수행시간이 약 15% 단축되었고, 오차율은 약 35% 줄어들었으며, 참정보 비율은 약 14% 증가되었다. Estimating the correct pose of augmented objects on the real camera view efficiently is one of the most important questions in image tracking area. In computer vision, Homography is used for camera pose estimation in augmented reality system with markerless. To estimating Homography, several algorithm like SURF features which extracted from images are used. Based on extracted features, Homography is estimated. For this purpose, RANSAC algorithm is well used to estimate homography and DCS-RANSAC algorithm is researched which apply constraints dynamically based on Constraint Satisfaction Problem to improve performance. In DCS-RANSAC, however, the dataset is based on pattern of feature distribution of images manually, so this algorithm cannot classify the input image, pattern of feature distribution is not recognized in DCS-RANSAC algorithm, which lead to reduce it`s performance. To improve this problem, we suggest the KCS-RANSAC algorithm using K-means clustering in CS-RANSAC to cluster the images automatically based on pattern of feature distribution and apply constraints to each image groups. The suggested algorithm cluster the images automatically and apply the constraints to each clustered image groups. The experiment result shows that our KCS-RANSAC algorithm outperformed the DCS-RANSAC algorithm in terms of speed, accuracy, and inlier rate.

온톨로지 기반의 수강지도 시스템

오경진(Kyeong-Jin Oh),윤의녕(Ui-Nyoung Yoon),조근식(Geun-Sik Jo) 한국지능정보시스템학회 2014 지능정보연구 Vol.20 No.2

Course guidance is a mentoring process which is performed before students register for coming classes. The course guidance plays a very important role to students in checking degree audits of students and mentoring classes which will be taken in coming semester. Also, it is intimately involved with a graduation assessment or a completion of ABEEK certification. Currently, course guidance is manually performed by some advisers at most of universities in Korea because they have no electronic systems for the course guidance. By the lack of the systems, the advisers should analyze each degree audit of students and curriculum information of their own departments. This process often causes the human error during the course guidance process due to the complexity of the process. The electronic system thus is essential to avoid the human error for the course guidance. If the relation data model-based system is applied to the mentoring process, then the problems in manual way can be solved. However, the relational data model-based systems have some limitations. Curriculums of a department and certification systems can be changed depending on a new policy of a university or surrounding environments. If the curriculums and the systems are changed, a scheme of the existing system should be changed in accordance with the variations. It is also not sufficient to provide semantic search due to the difficulty of extracting semantic relationships between subjects. In this paper, we model a course mentoring ontology based on the analysis of a curriculum of computer science department, a structure of degree audit, and ABEEK certification. Ontology-based course guidance system is also proposed to overcome the limitation of the existing methods and to provide the effectiveness of course mentoring process for both of advisors and students. In the proposed system, all data of the system consists of ontology instances. To create ontology instances, ontology population module is developed by using JENA framework which is for building semantic web and linked data applications. In the ontology population module, the mapping rules to connect parts of degree audit to certain parts of course mentoring ontology are designed. All ontology instances are generated based on degree audits of students who participate in course mentoring test. The generated instances are saved to JENA TDB as a triple repository after an inference process using JENA inference engine. A user interface for course guidance is implemented by using Java and JENA framework. Once a advisor or a student input student’s information such as student name and student number at an information request form in user interface, the proposed system provides mentoring results based on a degree audit of current student and rules to check scores for each part of a curriculum such as special cultural subject, major subject, and MSC subject containing math and basic science. Recall and precision are used to evaluate the performance of the proposed system. The recall is used to check that the proposed system retrieves all relevant subjects. The precision is used to check whether the retrieved subjects are relevant to the mentoring results. An officer of computer science department attends the verification on the results derived from the proposed system. Experimental results using real data of the participating students show that the proposed course guidance system based on course mentoring ontology provides correct course mentoring results to students at all times. Advisors can also reduce their time cost to analyze a degree audit of corresponding student and to calculate each score for the each part. As a result, the proposed system based on ontology techniques solves the difficulty of mentoring methods in manual way and the proposed system derive correct mentoring results as human conduct.

지식의 증류기법을 이용한 샷 경계 검출 모델

박성민(Sung Min Park),윤의녕(Ui Nyoung Yoon),조근식(Geun-Sik Jo) 한국방송·미디어공학회 2019 한국방송공학회 학술발표대회 논문집 Vol.2019 No.6

SSD 방법을 이용한 Cut transition 검출 모델

박성민(Sungmin Park),윤의녕(Ui Nyoung Yoon),조근식(Geun-Sik Jo) Korean Institute of Information Scientists and Eng 2020 정보과학회논문지 Vol.47 No.7

Shot boundary detection is constantly being studied as an essential technique for analyzing video content. In this paper, we propose an End-to-End Learning model using the SSD (Single Shot Multibox Detector) method to resolve the shortcomings of the existing research and to identify the exact location of the cut transition. We applied the concept of the Multi-Scale Feature Map and Default box of the SSD to predict multiple cut transitions, and combined the concept of Image Concatenation, one of the image comparison methods, with the model to reinforce the feature information of the cut transitions. The proposed model showed 88.7% and 98.0% accuracy in the re-labeled ClipShots and TRECVID 2007 datasets, respectively, compared to the latest research. Additionally, it detected a range closer to the correct answer than the existing deep learning model.

다양한 음성 특징값을 이용한 CNN 기반의 감정 인식 모델

이정현(Jung Hyun Lee),윤의녕(Ui Nyoung Yoon),조근식(Geun-Sik Jo) 한국정보과학회 2019 한국정보과학회 학술발표논문집 Vol.2019 No.6

RDGAN : 얼굴 감정인식 데이터 증강을 위한 데이터 재조정 GAN

김진용(Jin Yong Kim),윤의녕(Ui Nyoung Yoon),이명오(Myeong Oh Lee),조근식(Geun-Sik Jo) 한국정보과학회 2019 한국정보과학회 학술발표논문집 Vol.2019 No.12

효과적인 평면 호모그래피 추정을 위한 CS-RANSAC 기반의 특징점 필터링 방법

김대우 ( Dae Woo Kim ),윤의녕 ( Ui Nyoung Yoon ),조근식 ( Geun Sik Jo ) 한국정보처리학회 2016 정보처리학회논문지. 소프트웨어 및 데이터 공학 Vol.5 No.6

증강현실 분야에서 호모그래피(Homography)를 이용한 비마커 기반의 객체 추적 기술(Markerless tracking)은 카메라의 방향, 위치를 파악하여 실세계의 영상에 가상의 객체를 정확하고 자연스럽게 증강할 수 있는 기술이다. 이와 같은 호모그래피를 추정하기 위한 방법으로 RANSAC 알고리즘이 많이 사용되고 있으며, 최근 기존의 RANSAC 알고리즘에 제약 조건 문제(Constraint Satisfaction Problem)를 적용하여 정확도를 향상시키고, 처리시간을 줄인 CS-RANSAC 알고리즘에 대한 연구가 진행되고 있다. 하지만 CS-RANSAC 알고리즘은 샘플링 단계에서 정확도 가 낮은 호모그래피를 추정하게 하는 특징점이 선택되어 불필요한 연산으로 인해 알고리즘의 효율성이 저하되는 경우가 있다. 따라서 본 논문 에서는 효과적인 평면 호모그래피 추정을 위한 CS-RANSAC 기반의 특징점 필터링 방법을 제안한다. 제안하는 방법은 호모그래피 평가 단계에서 Symmetric Transfer Error로 정확도가 높은 호모그래피를 추정하게 하는 특징점인지를 평가하고 불필요한 특징점들을 다음 샘플링 단계에서 제외함으로써 정확도를 향상키고 처리시간을 줄였다. 제안하는 CS-RANSAC 기반의 특징점 필터링 방법의 성능평가를 위하여 제안하는 방법을 적용한 알고리즘과 기존의 RANSAC 알고리즘, CS-RANSAC 알고리즘의 수행시간과 오차율(Symmetric Transfer Error), 참정보 포함 비율을 비교하였다. 실험 결과 본 논문에서 제안한 방법이 기존 CS-RANSAC 알고리즘보다 수행시간이 평균적으로 약 5% 단축되었고 오차율 은 약 14% 줄어들어 더욱 정확한 호모그래피를 추정 할 수 있게 되었다. Markerless tracking for augmented reality using Homography can augment virtual objects correctly and naturally on live view of real-world environment by using correct pose and direction of camera. The RANSAC algorithm is widely used for estimating Homography. CS-RANSAC algorithm is one of the novel algorithm which cooperates a constraint satisfaction problem(CSP) into RANSAC algorithm for increasing accuracy and decreasing processing time. However, CS-RANSAC algorithm can be degraded performance of calculating Homography that is caused by selecting feature points which estimate low accuracy Homography in the sampling step. In this paper, we propose feature point filtering method based on CS-RANSAC for efficient planar Homography estimating the proposed algorithm evaluate which feature points estimate high accuracy Homography for removing unnecessary feature point from the next sampling step using Symmetric Transfer Error to increase accuracy and decrease processing time. To evaluate our proposed method we have compared our algorithm with the bagic CS-RANSAC algorithm, and basic RANSAC algorithm in terms of processing time, error rate(Symmetric Transfer Error), and inlier rate. The experiment shows that the proposed method produces 5% decrease in processing time, 14% decrease in Symmetric Transfer Error, and higher accurate homography by comparing the basic CS-RANSAC algorithm.

특징 연결과 깊이별 분리 컨볼루션을 이용한 효율적인 얼굴 감정인식 CNN

이명오(Myeong Oh Lee),윤의녕(Ui Nyoung Yoon),고승현(Seunghyun Ko),조근식(Geun-Sik Jo) 한국정보과학회 2019 한국정보과학회 학술발표논문집 Vol.2019 No.6

머신러닝 기법을 적용한 CS-RANSAC 알고리즘

고승현 ( Seunghyun Ko ),윤의녕 ( Ui-nyoung Yoon ),주마백 ( Jumabek Alikhanov ),조근식 ( Geun-sik Jo ) 한국정보처리학회 2016 한국정보처리학회 학술대회논문집 Vol.23 No.2

증강현실에서 영상과 증강된 콘텐츠 간의 이질감을 줄이기 위해서 정확한 호모그래피 행렬을 추정해야 하며, 정확한 호모그래피 행렬을 추정할때 RANSAC 알고리즘이 널리 사용된다. 그러나 RANSAC 알고리즘은 랜덤 샘플링 과정을 반복적으로 거치기 때문에 불필요한 연산 과정이 발생하고 이로 인해 알고리즘의 효율이 저하된다. 이러한 단점을 극복하기 위해 DCS-RANSAC 알고리즘이 제안 되었다. 제안된 DCS-RANSAC 알고리즘은 이미지를 특징점 분포 패턴에 따라 그룹으로 분류하고 각 그룹에 제약조건 문제를 적용하여 불필요한 연산 과정을 줄이고 정확도를 향상시킨 알고리즘이다. 그러나 DCS-RANSAC 알고리즘에서 사용된 이미지 그룹 데이터는 수동적인 방법을 통해 직관적으로 분류되어 있지만 특징점 분포 패턴이 다양하지 않아 분류시 정확도가 저하되는 경우가 있다. 위의 문제점을 해결하기 위해 본 논문에서는 머신러닝 기법을 통해 이미지들을 자동으로 분류하고 각 그룹마다 각기 다른 제약조건을 적용하는 MCS-RANSAC 알고리즘을 제안한다. 제안하는 알고리즘은 머신러닝 기법을 사용하여 전처리 단계에서 이미지를 분류하고 분류된 이미지에 제약조건을 적용시켜 알고리즘의 처리시간을 줄이고 정확도를 향상시켰다. 실험 결과 본 논문에서 제안하는 MCS-RANSAC은 DCS-RANSAC 알고리즘에 비해 수행시간이 약 6% 단축되었고 호모그래피 오차율은 약 15% 줄어들었으며 참정보 비율은 2.8% 증가한 것으로 확인되었다.