MDP-IoT: MDP based interest forwarding for heterogeneous traffic in IoT-NDN environment

Muralidharan, Shapna,Roy, Abhishek,Saxena, Navrati North-Holland 2018 Future generations computer systems Vol.79 No.3

<P><B>Abstract</B></P> <P>Internet of Things (IoT) a vision, being built today, holds a new rule for future “anything that can be connected will be connected”. IoT needs to support a multitude of heterogeneous objects extended with sensors, actuators, RFID’s, etc. These “Smart Objects” need unique identification, autonomous data transfer and communication with other objects. Consequently, these unique requisites of IoT need a promising future Internet architecture as it mostly revolves around data. Furthermore, the existing host-centric IP standards though advantageous, faces challenges like additional protocols for mobility, end-to-end security while deploying it with massive IoT applications. Named Data Networking (NDN) project is a new evolving data-centric internet architecture with innovative capabilities like caching, named data, security which mainly suits the specifications of IoT thereby proposed to solve the shortcomings of IP. NDN traditionally supports a PULL based traffic and its stateful forwarding engine despite its skillful nature need some modification while designing for an IoT system. In this paper, our foremost work is to classify and prioritize IoT traffic and enable delay-intolerant applications with low latency, to retrieve Data efficiently. Next, we propose a Markov Decision Process (MDP) based Interest scheduling for IoT traffic with varying priorities and measure the performance with different traffic probabilities. Our simulation results show that prioritizing and treating requests based on their traffic type can reduce network load by 30 % thereby improving QoS in an IoT-NDN environment. The MDP-based IoT model schedules’ the Interest to the best interface efficiently reducing the RTT values on an average of 20 % – 30 % than conventional forwarding strategies. The incurred delay is ∼ 30 % better than existing work and forwarding strategies.</P> <P><B>Highlights</B></P> <P> <UL> <LI> A Markov Decision Process (MDP)-based Interest Scheduling in IoT-NDN scenario. </LI> <LI> The model is proposed to satisfy delay-intolerant IoT applications efficiently. </LI> <LI> Prioritizing IoT traffic, and then scheduling the Interests with low latencies to right interfaces. </LI> <LI> This results in less RTT, thereby meeting latency requirements. </LI> <LI> Efficient model to solve Interest scheduling as the IoT system has many uncertainties. </LI> </UL> </P>

Markov Decision Process-based Potential Field Technique for UAV Planning

CHAEHWAN MOON,안재명 한국산업응용수학회 2021 Journal of the Korean Society for Industrial and A Vol.25 No.4

This study proposes a methodology for mission/path planning of an unmanned aerial vehicle (UAV) using an artificial potential field with the Markov Decision Process (MDP). The planning problem is formulated as an MDP. A low-resolution solution of the MDP is obtained and used to define an artificial potential field, which provides a continuous UAV mission plan. A numerical case study is conducted to demonstrate the validity of the proposed technique.

Optimal Energy Management Policy of Mobile Energy Gateway

Yang Zhang,Niyato, Dusit,Ping Wang,Dong In Kim IEEE 2016 IEEE Transactions on Vehicular Technology VT Vol.65 No.5

<P>With the advancement of wireless energy harvesting and transfer technologies, e.g., radio frequency (RF) energy, mobile nodes are fully untethered as energy supply is more ubiquitous. The mobile nodes can receive energy from wireless chargers, which can be static or mobile. In this paper, we introduce the use of a mobile energy gateway that can receive energy from a fixed charging facility, as well as move and transfer energy to other users. The mobile energy gateway aims to maximize the utility by optimally taking energy charging/transferring actions. We formulate the optimal energy charging/transferring problem as a Markov decision process (MDP). The MDP model is then solved to obtain the optimal energy management policy for the mobile energy gateway. Furthermore, the optimal energy management policy obtained from the MDP model is proven to have a threshold structure. We conduct an extensive performance evaluation of the MDP-based energy management scheme. The proposed MDP-based scheme outperforms several conventional baseline schemes in terms of expected overall utility.</P>

CG-E2S2: Consistency-guaranteed and energy-efficient sleep scheduling algorithm with data aggregation for IoT

Ko, Haneul,Lee, Jaewook,Pack, Sangheon Elsevier 2019 Future generation computer systems Vol.92 No.-

<P><B>Abstract</B></P> <P>In data acquisition (DAQ)-based services of Internet of things (IoT), IoT devices sense and transmit data to the application server through IoT gateway (GW). Due to the energy limitation of IoT devices, it is important to increase their energy efficiency. Further, when data from a very large number of IoT devices is individually transmitted, the data traffic volume can be significant. To resolve these issues, IoT devices and IoT GW can use sleep mode and data aggregation, respectively. However, when the IoT devices are in sleep mode for a long time and/or data are aggregated in IoT GW for a long time without any transmissions, data can become inconsistent. In this paper, we propose a consistency-guaranteed and energy efficient sleep scheduling algorithm (CG-E2S2) with data aggregation. In CG-E2S2, the optimal sleep duration of IoT devices and aggregation duration in IoT GW are jointly determined by means of a Markov decision process (MDP) with the consideration of energy efficiency of IoT devices, data traffic in networks, and data consistency. The evaluation results demonstrate that CG-E2S2 with the optimal policy outperforms the comparison schemes in terms of energy efficiency, data traffic volume, and data consistency.</P> <P><B>Highlights</B></P> <P> <UL> <LI> Sleep and aggregation durations are jointly optimized and determined by Markov decision process. </LI> <LI> Tradeoff between energy efficiency, traffic volume, and data consistency is investigated and optimized. </LI> <LI> Valuable guidelines for designing energy efficient IoT environments are provided. </LI> </UL> </P>

Industrial Internet of Things (IIoT) 환경에서 Risk-Level 기반 AoI를 고려한 효율적인 전송 기법

이재찬,정재현,박예본 한국혁신산업학회 2023 혁신산업기술논문지 Vol.1 No.1

재난 발생 시 이송 및 병원 자원 제약을 고려한 이송환자우선순위 결정 연구

신교홍,이태식 한국방재학회 2014 한국방재학회논문집 Vol.14 No.2

EMS resources management is one of the determinants to maximize the number of survivors effective first response in the aftermathof mass casualties. This paper concerns a problem of patient prioritization for EMS provision by constructing a Markov Decision Processes(MDP) model. While prior research tends to focus solely on transport resources (i.e., ambulance), we show that factors on theinvolved hospitals (i.e., capacity and capability) in the disaster response affect optimal response policies. Experiments on hypotheticalscenarios are conducted to compare the proposed model with the existing algorithms from the literature, including the standard triagepractice known as START(Simple Triage And Rapid Treatment). We show that considering hospital factors can save more patientsthan the other algorithms in most scenarios. The results of the study suggest that capacity and capability of the hospitals participatingin the response should be factored into decision makings in the EMS response to maximize the life savings. 재난으로 인해 다수 환자가 동시에 발생하는 경우, 효과적인 가용자원 운용을 위해서 고려해야할 요인으로 응급의료서비스 제공의 우선순위 결정과 해당 환자를 이송할 병원 선정이 있다. 이에 본 연구에서는 두 가지 요인을 동시에 결정하는 연구를 진행하였다. 각 병원의 서비스율과 중증 환자에 대한 치료 역량이 고려된 Markov Decision Processes 모델을 설계하여 기대 생존 환자 수를 최대화 하는최적 결정을 구했다. 모델로부터 얻어진 최적 결정은, 재난 현장 주변 병원의 진료 자원 여유 상태에 따라 우선적으로 이송해야하는 환자 집단이 달라짐을 보여준다. 계산된 최적 결정의 성능을 평가하기 위해 최근 관련 연구 문헌의 알고리즘 및 실제 재난 현장에서 일반적으로 사용되고 있는 START(Simple Triage And Rapid Treatment)방법과의 비교를 수행하였고, 타 알고리즘에 비해 대부분의 시나리오에서 더 많은 환자를 살릴 수 있다는 결과를 얻었다. 이러한 결과에 바탕하여, 발전된 재난사고 대처가 이루어지도록 재난 발생 시사고 현장 주변의 응급실 정보를 신속하게 공유하여 이송환자 우선순위를 결정하는 것을 제안한다.

Procurement scheduling under supply and demand uncertainty

Joohyun Shin,Jay H. Lee 제어로봇시스템학회 2015 제어로봇시스템학회 국제학술대회 논문집 Vol.2015 No.10

Supply chain of a manufacturing system contains procurement activity, and unloading raw materials from delivery vessels to storage tanks should be scheduled optimally, subject to the operational constraints. In general, an MILP model is used for a systematic procurement scheduling. However if there exists significant uncertainty in supply and demand, the solution obtained from the deterministic model may be suboptimal or even infeasible. Therefore in this study, two alternative approaches are formulated to consider these uncertainties: reactive rescheduling in the rolling horizon manner, and Markov decision process (MDP) formulation based scheduling that incorporates future uncertainty into the scheduling directly. In order to solve the MDP problem, algorithmic approximation strategies (such as approximate dynamic programming) are studied and applied for reducing computational challenges. Finally, their performances are compared with those of the original MILP model for a simple case study.

POMDP와 Exploration Bonus를 이용한 지역적이고 적응적인 QoS 라우팅 기법

한정수,Han Jeong-Soo 한국통신학회 2006 韓國通信學會論文誌 Vol.31 No.3b

In this paper, we propose a Localized Adaptive QoS Routing Scheme using POMDP and Exploration Bonus Techniques. Also, this paper shows that CEA technique using expectation values can be simply POMDP problem, because performing dynamic programming to solve a POMDP is highly computationally expensive. And we use Exploration Bonus to search detour path better than current path. For this, we proposed the algorithm(SEMA) to search multiple path. Expecially, we evaluate performances of service success rate and average hop count with $\phi$ and k performance parameters, which is defined as exploration count and intervals. As result, we knew that the larger $\phi$, the better detour path search. And increasing n increased the amount of exploration. 본 논문에서는 Localized Aptive QoS 라우팅을 위해 POMDP(Partially Observable Markov Decision Processes)와 Exploration Bonus 기법을 사용하는 방법을 제안하였다. 또한, POMDP 문제를 해결하기 위해 Dynamic Programming을 사용하여 최적의 행동을 찾는 연산이 매우 복잡하고 어렵기 때문에 CEA(Certainty Equivalency Approximation) 기법을 통한 기댓값 사용으로 문제를 단순하였으며, Exploration Bonus 방식을 사용해 현재 경로보다 나은 경로를 탐색하고자 하였다. 이를 위해 다중 경로 탐색 알고리즘(SEMA)을 제안했다. 더욱이 탐색의 횟수와 간격을 정의하기 위해 $\phi$와 k 성능 파라미터들을 사용하여 이들을 통해 탐색의 횟수 변화를 통한 서비스 성공률과 성공 시 사용된 평균 홉 수에 대한 성능을 살펴보았다. 결과적으로 $\phi$ 값이 증가함에 따라 현재의 경로보다 더 나은 경로를 찾게 되며, k 값이 증가할수록 탐색이 증가함을 볼 수 있다.

Optimal Stochastic Policies in a network coding capable Ad Hoc Networks

( Hayoung Oh ) 한국인터넷정보학회 2014 KSII Transactions on Internet and Information Syst Vol.8 No.12

Network coding is a promising technology that increases system throughput by reducing the number of packet transmissions from the source node to the destination node in a saturated traffic scenario. Nevertheless, some packets can suffer from end-to-end delay, because of a queuing delay in an intermediate node waiting for other packets to be encoded with exclusive or (XOR). In this paper, we analyze the delay according to packet arrival rate and propose two network coding schemes, iXOR (Intelligent XOR) and oXOR (Optimal XOR) with Markov Decision Process (MDP). They reduce the average delay, even under an unsaturated traffic load, through the Holding-χ strategy. In particular, we are interested in the unsaturated network scenario. The unsaturated network is more practical because, in a real wireless network, nodes do not always have packets waiting to be sent. Through analysis and extensive simulations, we show that iXOR and oXOR are better than the Distributed Coordination Function (DCF) without XOR (the general forwarding scheme) and XOR with DCF with respect to average delay as well as delivery ratio.

Data Delivery based on Mobile Roadside Unit and with QoS Guarantee for Vehicular Ad Hoc Networks

Li Chen,Zhijun Li,Shouxu Jiang 보안공학연구지원센터 2013 International Journal of Future Generation Communi Vol.6 No.5