Visual Presence: Viewing Geometry Visual Information of UHD S3D Entertainment

Oh, Heeseok,Lee, Sanghoon IEEE 2016 IEEE TRANSACTIONS ON IMAGE PROCESSING - Vol.25 No.7

<P>To maximize the presence experienced by humans, visual content has evolved to achieve a higher visual presence in a series of high definition (HD), ultra HD (UHD), 8K UHD, and 8K stereoscopic 3D (S3D). Several studies have introduced visual presence delivered from content when viewing UHD S3D from a content analysis perspective. Nevertheless, no clear definition has been presented for visual presence, and only a subjective evaluation has been relied upon. The main reason for this is that there is a limitation to defining visual presence via the use of content information itself. In this paper, we define the visual presence for each viewing environment, and investigate a novel methodology to measure the experienced visual presence when viewing both 2D and 3D via the definition of a new metric termed volume of visual information by quantifying the influence of the viewing geometry between the display and viewer. To achieve this goal, the viewing geometry and display parameters for both flat and atypical displays are analyzed in terms of human perception by introducing a novel concept of pixel-wise geometry. In addition, perceptual weighting through analysis of content information is performed in accordance with monocular and binocular vision characteristics. In the experimental results, it is shown that the constructed model based on the viewing geometry, content, and perceptual characteristics has a high correlation of about 84% with subjective evaluations.</P>

Enhancement of Visual Comfort and Sense of Presence on Stereoscopic 3D Images

Heeseok Oh,Jongyoo Kim,Jinwoo Kim,Taewan Kim,Sanghoon Lee,Bovik, Alan Conrad IEEE 2017 IEEE TRANSACTIONS ON IMAGE PROCESSING - Vol.26 No.8

<P>Conventional stereoscopic 3D (S3D) displays do not provide accommodation depth cues of the 3D image or video contents being viewed. The sense of content depths is thus limited to cues supplied by motion parallax (for 3D video), stereoscopic vergence cues created by presenting left and right views to the respective eyes, and other contextual and perspective depth cues. The absence of accommodation cues can induce two kinds of accommodation vergence mismatches (AVM) at the fixation and peripheral points, which can result in severe visual discomfort. With the aim of alleviating discomfort arising from AVM, we propose a new visual comfort enhancement approach for processing S3D visual signals to deliver a more comfortable 3D viewing experience at the display. This is accomplished via an optimization process whereby a predictive indicator of visual discomfort is minimized, while still aiming to maintain the viewer's sense of 3D presence by performing a suitable parallax shift, and by directed blurring of the signal. Our processing framework is defined on 3D visual coordinates that reflect the nonuniform resolution of retinal sensors and that uses a measure of 3D saliency strength. An appropriate level of blur that corresponds to the degree of parallax shift is found, making it possible to produce synthetic accommodation cues implemented using a perceptively relevant filter. By this method, AVM, the primary contributor to the discomfort felt when viewing S3D images, is reduced. We show via a series of subjective experiments that the proposed approach improves visual comfort while preserving the sense of 3D presence.</P>

Blind Deep S3D Image Quality Evaluation via Local to Global Feature Aggregation

Heeseok Oh,Sewoong Ahn,Jongyoo Kim,Sanghoon Lee IEEE 2017 IEEE TRANSACTIONS ON IMAGE PROCESSING - Vol.26 No.10

<P>Previously, no-reference (NR) stereoscopic 3D (S3D) image quality assessment (IQA) algorithms have been limited to the extraction of reliable hand-crafted features based on an understanding of the insufficiently revealed human visual system or natural scene statistics. Furthermore, compared with full-reference (FR) S3D IQA metrics, it is difficult to achieve competitive quality score predictions using the extracted features, which are not optimized with respect to human opinion. To cope with this limitation of the conventional approach, we introduce a novel deep learning scheme for NR S3D IQA in terms of local to global feature aggregation. A deep convolutional neural network (CNN) model is trained in a supervised manner through two-step regression. First, to overcome the lack of training data, local patch-based CNNs are modeled, and the FR S3D IQA metric is used to approximate a reference ground-truth for training the CNNs. The automatically extracted local abstractions are aggregated into global features by inserting an aggregation layer in the deep structure. The locally trained model parameters are then updated iteratively using supervised global labeling, i.e., subjective mean opinion score (MOS). In particular, the proposed deep NR S3D image quality evaluator does not estimate the depth from a pair of S3D images. The S3D image quality scores predicted by the proposed method represent a significant improvement over those of previous NR S3D IQA algorithms. Indeed, the accuracy of the proposed method is competitive with FR S3D IQA metrics, having similar to 91% correlation in terms of MOS.</P>

Stereoscopic 3D Visual Discomfort Prediction: A Dynamic Accommodation and Vergence Interaction Model

Heeseok Oh,Sanghoon Lee,Bovik, Alan Conrad IEEE 2016 IEEE TRANSACTIONS ON IMAGE PROCESSING - Vol.25 No.2

<P>The human visual system perceives 3D depth following sensing via its binocular optical system, a series of massively parallel processing units, and a feedback system that controls the mechanical dynamics of eye movements and the crystalline lens. The process of accommodation (focusing of the crystalline lens) and binocular vergence is controlled simultaneously and symbiotically via cross-coupled communication between the two critical depth computation modalities. The output responses of these two subsystems, which are induced by oculomotor control, are used in the computation of a clear and stable cyclopean 3D image from the input stimuli. These subsystems operate in smooth synchronicity when one is viewing the natural world; however, conflicting responses can occur when viewing stereoscopic 3D (S3D) content on fixed displays, causing physiological discomfort. If such occurrences could be predicted, then they might also be avoided (by modifying the acquisition process) or ameliorated (by changing the relative scene depth). Toward this end, we have developed a dynamic accommodation and vergence interaction (DAVI) model that successfully predicts visual discomfort on S3D images. The DAVI model is based on the phasic and reflex responses of the fast fusional vergence mechanism. Quantitative models of accommodation and vergence mismatches are used to conduct visual discomfort prediction. Other 3D perceptual elements are included in the proposed method, including sharpness limits imposed by the depth of focus and fusion limits implied by Panum's fusional area. The DAVI predictor is created by training a support vector machine on features derived from the proposed model and on recorded subjective assessment results. The experimental results are shown to produce accurate predictions of experienced visual discomfort.</P>

Deep Visual Discomfort Predictor for Stereoscopic 3D Images

Oh, Heeseok,Ahn, Sewoong,Lee, Sanghoon,Bovik, Alan Conrad IEEE 2018 IEEE TRANSACTIONS ON IMAGE PROCESSING - Vol.27 No.11

<P>Most prior approaches to the problem of stereoscopic 3D (S3D) visual discomfort prediction (VDP) have focused on the extraction of perceptually meaningful handcrafted features based on models of visual perception and of natural depth statistics. Toward advancing performance on this problem, we have developed a deep learning-based VDP model named deep visual discomfort predictor (DeepVDP). The DeepVDP uses a convolutional neural network (CNN) to learn features that are highly predictive of experienced visual discomfort. Since a large amount of reference data is needed to train a CNN, we develop a systematic way of dividing the S3D image into local regions defined as patches and model a patch-based CNN using two sequential training steps. Since it is very difficult to obtain human opinions on each patch, instead a proxy ground-truth label that is generated by an existing S3D visual discomfort prediction algorithm called 3D-VDP is assigned to each patch. These proxy ground-truth labels are used to conduct the first stage of training the CNN. In the second stage, the automatically learned local abstractions are aggregated into global features via a feature aggregation layer. The learned features are iteratively updated via supervised learning on subjective 3D discomfort scores, which serve as ground-truth labels on each S3D image. The patch-based CNN model that has been pretrained on proxy ground-truth labels is subsequently retrained on true global subjective scores. The global S3D visual discomfort scores predicted by the trained DeepVDP model achieve the state-of-the-art performance as compared with previous VDP algorithms.</P>

통계적 깊이 특성 기반의 3 차원 영상 시각 피로도 예측 기법

오희석(Heeseok Oh),김성현(Seonghyun Kim),이호재(Hojae Lee),이인웅(Inwoong Lee),김학섭(Haksub Kim),이상훈(Sanghoon Lee) 한국통신학회 2014 한국통신학회 학술대회논문집 Vol.2014 No.1

Metamaterial을 이용한 이중대역 발룬의 설계

오희석(Heeseok Oh),남상욱(Sangwook Nam) 대한전자공학회 2008 電子工學會論文誌-TC (Telecommunications) Vol.45 No.8

본 논문에서는 이중대역에서 작동하는 발룬을 제안하고 모의실험을 실시하였다. 기본적인 형태는 Wilkinson Power Divider에서 출발하여 각 단자(Port)간의 고립도(Isolation)를 향상시키기 위하여 λ/2 전송선을 단자 2와 3 사이에 삽입하였고, 이중대역 및 크기의 소형화를 위해서 λ/4 길이의 전송선을 metamaterial 구조인 CRLH(Composite Right/Left-Handed) 형태와 D-CRLH(Dual Composite Right/Left-Handed) 형태를 사용하여 TDMB 주차수대역인 195㎒, 그리고 DVB-H 주파수대역인 670㎒에서 작동하는 발룬을 설계하였다. 제안된 구조로 설계된 발룬의 반사손실(return loss)에 대한 최소값은 -12.98㏈(S11)이며, 고립도(isolation)는 최소 -12.4㏈, 그리고 출력신호간의 불균형은 0.08㏈보다 작고, 위상오차는 최대 2.8°이다. This paper proposes a dual-band balun which is based on Wilkinson power divider. By inserting λ/2 transmission line between port 2 and 3, this balun shows good matching at all ports and improved isolation. We use matamaterial(CRLH, D-CRLH) structure for a miniaturization of the circuit implementation and dual-band operation at TDMB frequency range(195㎒) and DVB-H frequency range(670㎒). The proposed balun is designed with return loss larger than -12.98㏈ at all port, and isolation larger than -12.4㏈, the amplitude imbalance between output signals less than 0.08㏈, also phase differences of outputs less than 2.8°.

유사도 추정 기반 플렌옵틱 영상 내 단일 객체 추적 기술

오희석(Heeseok Oh) 대한전자공학회 2021 전자공학회논문지 Vol.58 No.2

단일 객체 추적은 컴퓨터비전의 오래된 기술 분야로써 감시, 국방 및 자율주행을 비롯한 다양한 응용 기술에 활용된다. 최근의 2차원 영상 내 객체 추적 기술들은 Siamese 구조의 심층신경망을 통해 추출된 타겟 객체와 탐색 영역의 특징 간 유사도를 추정함으로써 이루어진다. 이를 통해 추적의 신뢰성과 실시간성 측면에서 전통적인 필터 기반의 객체 추적 기술 대비 비약적인 성능 향상을 이루었으나 여전히 occlusion 발생 시 객체 추적의 빈번한 실패는 명확한 해결방안을 찾기 어려운 고난이도의 기술적 문제점으로 지적되어왔다. 이에 본 논문에서는 플렌옵틱 영상 기반으로 그 특성을 적극 활용하여 occlusion 발생에도 강건한 성능을 보장하는 객체 추적 알고리즘을 제안한다. 복수의 카메라를 통해 렌더링 된 플렌옵틱 영상은 포컬스택으로 표현되며, 서로 다른 초점 영역을 나타내는 다수의 포컬플레인으로 구성된다. 일반적인 2차원 영상과는 달리 플렌옵틱 영상의 포컬스택은 occlusion 발생 시에도 소수의 특정 초점 영역에서 타겟 객체의 추적을 위한 외형 정보를 포함하며 추적의 성공가능성이 존재한다. 따라서 본 논문에서는 해당 정보의 활용을 위해 Siamese 신경망을 통해 추출된 타겟 객체와 포컬플레인 영상의 고차원 특징 간 유사도를 추정함으로써 객체를 추적하는 플렌옵틱 객체 추적 모델을 구현하였다. 또한, 다수의 포컬플레인 입력으로 인한 오류를 최소화하고자 프레임별로 탐색 영역을 제한하는 알고리즘을 제안한다. 실제 플렌옵틱 영상에 제안하는 알고리즘 적용 시, occlusion 발생의 경우에도 기존 2차원 객체 추적 기술 대비 향상된 성능의 객체 추적이 가능함을 실험적으로 확인하였다. Single object tracking is one of the conventional fields in computer vision, and which is being employed by various applications including surveillance, defense, and autonomous driving. Recent 2D object tracking techniques adopt a similarity estimation between the extracted features from a target object and search regions by feeding them into a Siamese network. Such deep learning based object tracking methods have led the improved performances regarding both robustness and real-time capability, however, tracking the partially or even fully occluded object is challenging, and which is still remained as an insurmountable technical huddle in the related field. In order to resolve this problem, we introduce the novel plenoptic object tracking method guaranteeing the reliable performance when occlusion occurs. A focal stack can be rendered by plenoptic imaging with the calibrated multiple cameras, and which consists of several focal planes representing different focus regions. Differ to general 2D sequences, some of focal planes in a focal stack provide weak appearance information to track the target object stemmed from disparities of the separately located cameras. Thus, we utilize such characteristics to track an object in plenoptic sequences by estimating similarity between the features of target object and focal planes in hyperspace which are captured by a weight-sharing structured network. Additionally, towards minimizing the plenoptic object tracking error mainly caused by an exhaustive search over all focal planes, the adaptive search region restriction algorithm is also proposed. Through applying the proposed plenoptic object tracking scheme, the results show that promising performance can be achieved when even a target object is invisible.

IoT 디바이스 전송 주기 제어 및 관리 프레임워크 설계 및 구현

오희석(Heeseok Oh),김지형(Jeehyeong Kim),최원기(Wongi Choi),송민환(Minhwan Song),이상신(Sangshin Lee) 대한전자공학회 2022 대한전자공학회 학술대회 Vol.2022 No.11

이미지 기반 적대적 사례 생성 기술 연구 동향

오희석(Heeseok Oh) 한국정보보호학회 2020 情報保護學會誌 Vol.30 No.6

다양한 응용분야에서 심층신경망 기반의 학습 모델이 앞 다투어 이용됨에 따라 인공지능의 설명 가능한 동작 원리 해석과, 추론이 갖는 불확실성에 관한 분석 또한 심도 있게 연구되고 있다. 이에 심층신경망 기반 기계학습 모델의 취약성이 수면 위로 드러났으며, 이러한 취약성을 이용하여 악의적으로 모델을 공격함으로써 오동작을 유도하고자 하는 시도가 다방면으로 이루어짐에 의해 학습 모델의 강건함 보장은 보안 분야에서의 쟁점으로 부각되고 있다. 모델 추론의 입력으로 이용되는 이미지에 교란값을 추가함으로써 심층신경망의 오분류를 발생시키는 임의의 변형된 이미지를 적대적 사례라 정의하며, 본 논문에서는 최근 인공지능 및 컴퓨터비전 분야에서 이루어지고 있는 이미지 기반 적대적 사례의 생성 기법에 대하여 논한다.