Future landscape visualization using a city digital twin: integration of augmented reality and drones with implementation of 3D model-based occlusion handling

다국어 초록 (Multilingual Abstract)

The integration of augmented reality and drones allows past and future landscapes to be visualized from an aerial perspective. However, these visualizations still suffer from the occlusion problem, where the three-dimensional (3D) virtual model displayed in the real world is in front of a real-world object. Currently, city digital twins are essential for the sustainable development of cities and the development of detailed 3D models of cities. By visualizing the city digital twin, augmented reality can facilitate the participation of nonexpert citizens in the decision-making process of urban design, but research examples are limited. Here, using detailed city 3D models, we develop a digital-twin approach to outdoor augmented reality with occlusion handling for both first-person and bird’s-eye views. In a verification experiment, the occlusion handling accuracy of the prototype system was measured to be about 0.8 using intersection over union. The frame rate of the entire prototype system was about 30 fps, and the delay between the controller and the augmented reality device was about 3 s. The internet-based system architecture was developed to integrate augmented reality and drone systems. Our system allows multiple stakeholders involved in building construction projects to observe aerial perspectives of those projects, both on-site and off-site via an internet browser, using augmented reality with occlusion handling.

참고문헌 (Reference)

1 Tomkins, A., "Where the wild things will be : Adaptive visualisation with spatial computing" 6 : 140-147, 2021

2 Zollmann, S., "Visualization techniques in augmented reality : A taxonomy, methods and patterns" 27 (27): 3808-3825, 2020

3 Wiberg, A. H., "Visualisation of KPIs in zero emission neighbourhoods for improved stakeholder participation using virtual reality" 323 : 012074-, 2019

4 Dembski, F., "Urban digital twins for smart cities and citizens : The case study of Herrenberg, Germany" 12 (12): 1-17, 2020

5 Motlagh, N. H., "UAV-Based IoT platform : A crowd surveillance use case" 55 (55): 128-134, 2017

6 Jabbar, A., "Training deep neural networks for detecting drinking glasses using synthetic images" 354-363, 2017

7 Schrotter, G., "The digital twin of the city of Zurich for urban planning" 88 : 99-112, 2020

8 Goudarznia, T., "Testing the effectiveness of augmented reality in the public participation process : A case study in the city of Bernburg" 2 : 244-251, 2017

9 Westoby, M. J., "Structure-from-Motion’ photogrammetry : A low-cost, effective tool for geoscience applications" 179 : 300-314, 2012

10 Ruohom¨ aki, T., "Smart city platform enabling digital twin" 155-161, 2018

11 Virtual Singapore, "Singapore Government"

12 Chen, J. Y. C., "Review of low frame rate effects on human performance" 37 : 1063-1076, 2007

13 Dou, S. Q. Q., "Research on construction of spatio-temporal data visualization platform for GIS and BIM fusion" 42 : 555-563, 2019

14 Li, W., "Real-Time location-based rendering of urban underground pipelines" 7 (7): 1-17, 2018

15 Ham, Y., "Participatory sensing and digital twin city : Updating virtual city models for enhanced riskinformed decision-making" 36 (36): 1-12, 2020

16 Offenhuber, D., "Over the rainbow: Information design for low-resolution urban displays" Association for Computing Machinery 40-47, 2014

17 Kasperi, J., "Occlusion in outdoor augmented reality using geospatial building data" 30 : 1-10, 2017

18 Roxas, M., "Occlusion handling using semantic segmentation and visibility-based rendering for mixed reality" 20 : 1-8, 2018

19 Open Geospatial Consortium, "OGC city geography markup language (CityGML) 3.0 conceptual model users guide, 20-066, Version 3.0"

20 Elloumi, M., "Monitoring road traffic with a UAV-based system" 1-6, 2018

21 Haynes, P., "Mobile augmented reality for flood visualization" 109 : 380-389, 2018

22 Evangelidis, K., "Mixed reality : A reconsideration based on mixed objects and geospatial modalities" 11 (11): 2417-, 2021

23 PLATEAU, "Ministry of Land, Infrastructure, Transport and Tourism"

24 "Microsoft HoloLens"

25 Kikuchi, N, "Landscape visualization by integrating augmented reality and drones with occlusion handling to link real and virtual worlds – Towards city digital twin realization" 2 : 521-528, 2021

26 Fast.com, "Internet Speed Test"

27 Tomkins, A., "Interactive landscape design and flood visualisation in augmented reality" 3 (3): 1-13, 2019

28 Liu, D., "Integrating building informationmodel and augmented reality for drone-based building inspection" 35 : 1-40, 2021

29 Yan, L, "Integrating UAV development technology with augmented reality toward landscape tele-simulation" 1 : 423-432, 2019

30 Tian, Y., "Handling occlusions in augmented reality based on 3D reconstruction method" 156 : 96-104, 2015

31 Unal, M., "Geo-location based augmented reality application for cultural heritage using drones" 1-4, 2018

32 Koch, V, "Flying augmented reality: Supporting planning and simulation analysis by combining mixed reality methods using multicopter and pattern recognition" 1 : 843-849, 2011

33 Holynski, A., "Fast depth densification for occlusion-aware augmented reality" 37 (37): 1-11, 2018

34 Kim Jung In ; Li Sining ; Chen Xingbin ; Keung Calvin ; Suh Minjae ; Kim Tae Wan, "Evaluation framework for BIM-based VR applications in design phase" 한국CDE학회 8 (8): 910-922, 2021

35 Gallacher, D., "Drone applications for environmental management in urban spaces : A review" 3 (3): 1-14, 2017

36 Unal, M., "Distant augmented reality : Bringing a new dimension to user experience using drones" 17 : 1-12, 2020

37 Batty, M., "Digital twins" 45 (45): 817-820, 2018

38 Grieves, M, "Digital twin: Manufacturing excellence through virtual factory replication" 1-7, 2014

39 Mohammadi, N., "Devising a game theoretic approach to enable smart city digital twin analytics" 1995-2002, 2019

40 Du, R., "DepthLab : Real-time 3D interaction with depth maps for mobile augmented reality" 829-843, 2020

41 Valentin, J., "Depth from motion for smartphone AR" 37 (37): 1-19, 2018

42 Shahat, E., "City digital twin potentials : A review and research agenda" 13 : 1-20, 2021

43 Wen, M., "Augmented reality and unmanned aerial vehicle assist in construction management" 1570-1577, 2014

44 Kido, D., "Assessing future landscapes using enhanced mixed realitywith semantic segmentation by deep learning" 48 : 101281-, 2021

45 Tomohiro Fukuda ; Kazuki Yokoi ; Nobuyoshi Yabuki ; Ali Motamedi, "An indoor thermal environment design system for renovation using augmented reality" 한국CDE학회 6 (6): 179-188, 2019

46 Amazon, "Amazon prime air"

47 Gimeno, J., "Addressing the occlusion problem in augmented reality environments with phantom hollow objects" 21-24, 2018

48 Milgram, P., "A taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems"

49 Deng, T., "A systematic review of a digital twin city : A new pattern of urban governance toward smart cities" 6 (6): 125-134, 2021

50 Azuma, R. T, "A survey of augmented reality" 6 (6): 355-385, 1997

51 Hong, I., "A range-restricted recharging station coveragemodel for drone delivery service planning" 90 : 198-212, 2018

52 White, G, "A digital twin smart city for citizen feedback" 110 : 1-11, 2021

53 Herman, L., "A comparison of monoscopic and stereoscopic 3D visualizations : Effect on spatial planning in digital twins" 13 (13): 2976-, 2021