(A) study on non-contact biosignal measurement and diagnosis using facial skin images

국문 초록 (Abstract)

본 논문에서는 얼굴 피부 영상을 이용한 비접촉 생체신호 측정 및 진단 방법을 제안한다. 이 논문은 네 가지 세부 연구 내용을 가지고 있다.
세부 연구 내용을 제안하기에 앞서, 조명 변화 및 신체 떨림 등으로 인해 발생하는 여러 노이즈를 제거하기 위해 얼굴 검출 안정화를 수행하고, RGB뿐만 아니라 YCgCo, YCbCr 등의 색차 성분을 이용한다. 또한, 제안된 방법을 통해 생체신호 측정 시스템의 성능이 향상된다는 것을 확인하였다.
첫째, 피부 관련 생체신호 측정 및 진단 방법을 제안한다. 제안된 접근법은 얼굴 영상의 피부 관심 영역에서 계산된 색상 및 질감 데이터를 이용하여 피부 수분, 피부 유분, 피부 산도 및 피부 온도를 측정하며, 얼굴 피부 영상을 이용하여 계산된 맥박수, 호흡수 및 피부 온도를 이용하여 맥박수, 호흡수 및 혈압을 포함하는 활력징후의 필수 요소인 체온을 측정한다.
둘째는 심혈관 관련 생체신호 측정 및 진단 방법을 제안한다. 제안 방법은 다음과 같이 요약한다. (1) 혈당, 헤모글로빈은 피부 관심 영역에서 계산된 색상 데이터를 이용하여 측정한다. (2) 얼굴 영상의 피부 관심 영역에서 계산된 Cg 색상 신호로부터 산출된 맥파를 이용하여 맥박수, 기이맥, 부정맥 징후, 혈액점도, 혈량, 혈관 나이 및 혈과 탄성도를 측정한다.
셋째, 호흡기 관련 생체신호 측정 및 진단 방법을 제공하며, 얼굴 영상의 피부 관심 영역에서 계산된 Cg 색상 신호로부터 산출된 호흡 신호를 이용하여 호흡수, 폐기능(강제 호기량, 최대 호기량), 폐활량, 산소포화도를 측정한다.
넷째는 정신 건강 관련 생체신호 측정 및 진단 방법을 제안한다. 얼굴 영상에서 산출된 맥파와 호흡 신호를 분석하여 스트레스 및 우울증 지수를 측정하며, 전두엽 피부 관심 영역에서 계산된 Cg 색상 신호의 주파수 성분 분석을 통해 스트레스, 우울증 및 치매 지수를 측정한다. 제안된 주요 테마를 포함하는 얼굴 피부 영상을 이용한 비접촉 통합 생체신호 측정 및 진단 시스템 구축이 가능하다.

다국어 초록 (Multilingual Abstract)

This dissertation proposes a non-contact biosignal measurement and diagnosis method using facial skin images. The dissertation is organized around four main themes. Prior to proposing these themes, face detection stabilization is conducted, and color difference components such as YCgCo, YCbCr, etc. as well as RGB colors in region of interest for facial skin are used to eliminate noise caused by illumination changes and body tremors. It was confirmed that implementing these processes improves the performance of biosignal measurement.
First, this dissertation propose a skin surface biosignal measurement and diagnosis method that utilizes facial skin images. This approach employs color and texture data, calculated from the region of interest in the facial video captured by a camera, to measure skin moisture, skin oil, skin pH, and skin temperature. Additionally, using pulse rate, respiratory rate, and skin temperature calculated from facial skin images, this dissertation suggests a measuring method for body temperature, which is an essential factor of vital signs, including pulse rate, respiratory rate, and blood pressure.
Second, this dissertation suggest a cardiovascular biosignal measurement and diagnosis method that utilizes facial skin images. The approach is summarized as follows: (1) Blood glucose and hemoglobin are measured using various color data, calculated from the region of interest on the skin. (2) The pulse wave, extracted from the Cg color signal, is used to measure the pulse rate, paradoxical pulse, blood pressure, blood viscosity, blood volume and blood vessel elasticity.
Third, this dissertation provide a respiratory biosignal measurement and diagnosis method using facial skin images. The respiratory signal, extracted from the Cg color signal, is used to measure respiratory rate, lung function (such as forced vital capacity) and peak expiatory flow rate, lung capacity, oxygen saturation.
Fourth, this dissertation propose a mental biosignal measurement and diagnosis method using facial skin images. The stress and depression disorder index can be measured by analyzing the pulse wave and respiratory signal, and stress, depression disorder and dementia index can also be measured by analyzing the Cg color signal calculated from the skin region of interest in the frontal lobe. A non-contact integrated biosignal measurement and diagnosis system can be built using facial skin images with the proposed main themes.

목차 (Table of Contents)

• 1. Introduction 1
• 2. Related works 9
• 2.1 Skin Surface-Related Biosignals Measurement 15
• 2.2 Cardiovascular-Related Biosignals Measurement 19
• 2.3 Respiratory-Related Biosignals Measurement 31
• 2.4 Mental-Related Biosignals Measurement 40
• 3. Pre-processing for Biosignal Measurement 47
• 3.1 Face Detection Stabilization 47
• 3.2 Optimal ROI Determination 53
• 3.3 Color System Conversion 55
• 4. Biosignal Measurement and Diagnosis using Facial Skin Images 59
• 4.1 Skin Surface Biosignal Measurement and Diagnosis 67
• 4.1.1 Skin Moisture Measurement Method 67
• 4.1.2 Skin Oil Measurement Method 68
• 4.1.3 Skin pH Measurement Method 69
• 4.1.4 Skin Temperature Measurement Method 70
• 4.1.5 Body Temperature Measurement Method 72
• 4.1.6 Skin Surface Biosignal Diagnosis 73
• 4.2 Cardiovascular Biosignal Measurement and Diagnosis 74
• 4.2.1 Pulse wave Extraction Method 75
• 4.2.2 Pulse Rate Measurement Method 79
• 4.2.3 Deep Learning based Pulse Rate Measurement Method 81
• 4.2.4 Blood Pressure Measurement Method 82
• 4.2.5 Paradoxical Pulse Measurement Method 91
• 4.2.6 Blood Volume Measurement Method 93
• 4.2.7 Hemoglobin Measurement Method 94
• 4.2.8 Blood Glucose Measurement Method 96
• 4.2.9 Blood Viscosity Measurement Method 98
• 4.2.10 Cardiovascular Biosignal Diagnosis 99
• 4.3 Respiratory Biosignal Measurement and Diagnosis 101
• 4.3.1 Respiratory Signal Extraction Method 101
• 4.3.2 Respiration Rate Measurement Method 104
• 4.3.3 Lung Function Measurement Method 106
• 4.3.4 Oxygen Saturation Measurement Method 111
• 4.3.5 Respiratory Biosignal Diagnosis 113
• 4.4 Mental Biosignal Measurement and Diagnosis 114
• 4.4.1 Stress Index Measurement Method 115
• 4.4.2 Depression Index Measurement Method 121
• 4.4.3 Dementia Index Measurement Method 125
• 4.4.4 Mental Biosignal Diagnosis 126
• 5. Experiments and Results 127
• 5.1 Skin Surface Biosignal Measurement Experiment 128
• 5.2 Cardiovascular Biosignal Measurement Experiment 130
• 5.3 Respiratory Biosignal Measurement Experiment 148
• 5.4 Mental Biosignal Measurement Experiment 157
• 5.5 Integrated-Biosignal Measurement Application 170
• 6. Conclusions 173
• References 178
• Korean Abstract 200

참고문헌 (Reference)

1. 201, National Institute of Food and Drug Safety Evaluation of Ministry of Food and Drug Safety, Guidelines for approval and examination for each medical device item of UHealthcare, , 2016

2. 202, National Institute of Food and Drug Safety Evaluation of Ministry of Food and Drug Safety, Guidelines for Accuracy Assessment of Electronic Blood Pressure Meter, , 2007

3. 203, National Institute of Food and Drug Safety Evaluation of Ministry of Food and Drug Safety, Field Automatic Hemoglobin Meter Safety and Performance Test Method Guidelines, , 2017

4. 204, National Institute of Food and Drug Safety Evaluation of Ministry of Food and Drug Safety, Guidelines for Standard Test Methods for Personal Blood glucose measurement systems, , 2016

5. 205, National Institute of Food and Drug Safety Evaluation of Ministry of Food and Drug Safety, Guidelines for Performance Evaluation of Diagnostic Lung Capacity Meter, , 2011

6. Spirometers, De J. F., Breathe, 4.3, 251254, , 2008

7. Wood's lamp, Gupta, Lalit Kumar and, M. K. Singhi, 70, 131135, , 2004

8. Heart rate variability, van RavenswaaijArts et al, 118.6, 436447, , 1993

9. How should we define health?, Huber, Machteld. et al, 343, , 2011

10. Intelligent Pillow for Heart rate monitor, Yin, E. C. H. et al, 3.6, 791796, , 2013

11. Ethical problems of smart wearable devices, Chang, V. et al, 4th International Conference on Complexity, Future Information Systems and Risk, SciTePress, 121129, , 2019

12. Stress index estimation method using image, Hong, K. Park, J., No. 1018236580000, , 2018

13. A new method for blood viscosity measurement, Kim, S. et al, 94.1, 4756, , 2000

14. Respiration rate monitoring methods: A review, ALKhalidi, F. Q. et al, 46.6, 523529, , 2011

15. Method of measuring skin Ph level using images, Hong, K., Park, J., No. 1019285010000, , 2018

16. Cardiovascular disease and severe mental illness, Holt, Richard IG, Comorbidity of Mental and Physical Disorders, 179, 5465, , 2015

17. Method of measuring skin oil Degree using images, Hong, K., Park, J., No. 1018815570000, , 2018

18. Method of estimating skin temperature using images, Park, J., Hong, K., No. 101854246000, , 2018

19. Videobased heart rate measurement from human faces, Pursche, T. J. K. et al, 2012 IEEE international conference on consumer electronics (ICCE) , 544545, , 2012

20. Depression index estimation method using skin image, J., Hong, K. Park, No. 1022430170000, , 2022

21. Method and apparatus for measuring pulsus paradoxus, Arnold, Donald., U. S. Patent, No. 6,869,402, , 2005

22. Remote plethysmographic imaging using ambient light, Verkruysse, W. L. O. S., 16.26, 2143421445, , 2008

23. Seeing red: PPG biometrics using smartphone cameras, Lovisotto, Giulio, et al, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 818819, , 2020

24. Measurement of heart rate using photoplethysmography, Saquib, N. et al, 2015 International Conference on Networking Systems and Security (NSysS) , 16, , 2015

25. Method of measuring skin moisture level using images, Hong, K., No. 1019924030000, , 2019

26. A Study on the assessment of stress using wireless ECG, Lim, C. Y. et al, 16.2, 1723, , 2011

27. Smartphone sensors for health monitoring and diagnosis, Majumder, Sumit and, M. Jamal Deen, 19.9, 2164, , 2019

28. The syndrome of alternating bradycardia and tachycardia, Short, D. S., 16.2, 208, , 1954

29. Photoelectric plethysmography of the nasal septum in man, Hertzman, A. B., 37.2, 290292, , 1937

30. Cardiovascular risk factors for people with mental illness, Davidson, Sandra et al, 35.2, 196202, , 2001

31. Method and device for measuring blood viscosity using image, Hong, K., Park, J., No. 1019890500000, , 2019

32. 168 Predict Daily Life Stress based on Heart Rate Variability, Park, J., A dissertation for the degree of Master of Science submitted to the Graduate UNIST, , 2020

33. A device for skin moisture and environment humidity detection, Huang, TingHsiang, et al, 134.1, 206212, , 2008

34. Revise: Remote vital signs measurement using smartphone camera, Qiao, D. et al, 10, 131656131670, , 2022

35. An extended set of haarlike features for rapid object detection, Lienhart, R. et al, Proceedings. international conference on image processing, Vol. 1, 11, , 2002

36. Real Time Tracking of Heart Rate from Facial Video Using Webcam, P. R. et al, Oviyaa, M., 2020 Second International Conference on Inventive Research in Computing Applications (ICIRCA) , 17, , 2020

37. Wood’s lamp in dermatology: Applications in the daily practice, Veasey, John Verrinder, et al, 9.4, 3246, , 2017

38. Method and apparatus for measuring blood pressure using skin image, Park, J., Hong, K., KR Patent, No. 1022846710000, , 2021

39. ECGbased heartbeat classification for arrhythmia detection: A survey, Luz, E. J. D. S. et al, 127, 144164, , 2016

40. Method and apparatus for estimating blood volume based on skin image, Hong, K., Park, J., KR Patent, No. 1021329250000, , 2020

41. A novel nonintrusive decision support approach for heart rate measurement, Fernandes, S. L. et al, 139, 148156, , 2020

42. Remote heart rate measurement from face videos under realistic situations, Li, Xiaobai. et al, Proceedings of the IEEE conference on computer vision and pattern recognition, 42644271, , 2014

43. Depression recognition using remote photoplethysmography from facial videos, Casado, Constantino Álvarez, et al, IEEE Transactions on Affective Computing (, , 2023

44. Application of regression techniques with their advantages and disadvantages, Iqbal, Muhammad Ahmad, 4, 1117, , 2021

45. Deeplearningbased stress recognition with spatialtemporal facial information, Jeon, Taejae, et al, 21.22, 7498, , 2021

46. Management of Psychiatric Disorders in Patients with Cardiovascular Diseases, Gundugurti, P. Rao et al, 64. Suppl 2, S355, , 2022

47. Infrared camera assessment of skin surface temperature–effect of emissivity, Bernard, Vladan, et al, 583591, 29.6, , 2013

48. Noncontact estimation of heart rate and oxygen saturation using ambient light, Bal, U., 6.1, 8697, , 2015

49. Photoplethysmography and its application in clinical physiological measurement, Allen, J., Physiological measurement, 28.3, R1, , 2007

50. Method and apparatus analyzing dementia severity using froontal lobe skin image, Hong, K. Park, J., KR Patent, No. 1024878270000, , 2023

51. Method of estimating robust pulse rate and respiration rate using facial images, Hong, K., Park, J., KR Patent, No. 1023583250000, , 2022

52. A Featurebased Approach on Contactless Blood Pressure Estimation from Video Data, Wuerich, C. et al, 2022 30th European Signal Processing Conference (EUSIPCO) , 13431347, , 2022

53. Introducing contactless blood pressure assessment using a high speed video camera, Jeong, I. et al, 40, 110, , 2016

54. Method and apparatus for measuring robust continuous blood sugar using skin image, Hong, K., Park, J., KR Patent, No. 1024222810000, , 2022

55. Estimation of respiratory rate from thermography using respiratory likelihood index, Takahashi, Y. et al, 21.13, 4406, , 2021

56. A noncontact integrated bodyambient temperature sensors platform to contrast COVID19, Costanzo, Sandra, et al, Electronics, 9.10, 1658, , 2020

57. Smartphonebased blood pressure measurement using transdermal optical imaging technology, Luo, H. et al, Circulation: Cardiovascular Imaging, 12.8, e008857, , 2019

58. Measurement of skin hydration with a portable device and comparison with the Corneometer, Westermann, T. VA. et al, Skin Research and Technology, 26.4, 571576, , 2020

59. Remote heart rate measurement using lowcost RGB face video: a technical literature review, Rouast, P. V. et al, Frontiers of Computer Science, 12, 858872, , 2018

60. Methods and apparatuses for estimating robust body temperature using biosignal of skin image, Hong, K., Park, J., KR Patent, No. 1023994530000, , 2022

61. Differences in Hemoglobin Levels as Measured by Blood Gas and Auto Blood Cell Count Analyzers, Yoon, S, et al, 20.3, 242246, , 2009

62. The use of infrared thermography to detect the skin temperature response to physical activity, Tanda, Giovanni, J ournal of Physics: Conference Series, Vol. 655. No. 1. IOP Publishing, , 2015

63. Realtime mental stress detection using multimodality expressions with a deep learning framework, Zhang, Jing, et al, 16, 947168, , 2022

64. Cardiovascular, respiratory, and related disorders: key messages from Disease Control Priorities, Prabhakaran, Dorairaj. et al, 391.10126, 12241236, , 2018

65. Evaluation of thermometers for ear temperature measurement at the wards in a university hospital, Nascimento, Vitor. R. D., Diss. Faculdade de Ciências e Tecnologia, , 2012

66. Automatic Identification of Depression Using Facial Images with Deep Convolutional Neural Network, Kong, Xinru, et al, Medical Science Monitor: International Medical Journal of Experimental and Clinical Research, 28, e9364091, , 2022

67. Paradoxical pulse detection method using image and paradoxical pulse detection apparatus using image, Hong, K. Park, J., KR Patent, No. 1018692250000, , 2018

68. Methods and apparatuses for measuring and managing stress index using respiratory signals of skin image, Park, J., Hong, K., KR Patent, No. 1023526290000, , 2022

69. Automated recognition of major depressive disorder from cardiovascular and respiratory physiological signals, Zitouni, M. Sami et al, 13, 970993, , 2022

70. Deep 3DCNN for depression diagnosis with facial video recording of selfrating depression scale questionnaire, Xie, Wanqing, et al, 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) . IEEE, , 2021

71. Towards a noncontact method for identifying stress using remote photoplethysmography in academic environments, MoralesFajardo, Hector Manuel et al, 22.10, 3780, , 2022

72. Evaluation of vascular wall elasticity of human digital arteries using alternating currentsignal photoplethysmography, Uangpairoj, Pichitra. et al, 283295, , 2013

73. Design Of NonContact Human Body Temperature Detection Based On Internet Of Things (Iot) To Open The Door Automatically, Karsono, Kundang, et al, International J ournal of Science, Technology & Management, 3.4, 10121017, , 2022

74. Health promotion policy development in accordance with the paradigm shift from curative measures to preventive measures, Choi, E. J. et al, Seoul: Korea Institute for Health and Social Affairs, 10021, , 2011

75. Noncontact body temperature measurement: uncertainty evaluation and screening decision rule to prevent the spread of COVID19, Dell’Isola, Giovanni Battista et al, 21.2, 346, , 2021

76. Economic Report of the President Transmitted February 2010 Together with the Annual Report of the Council of Economic Advisors, Council of Economic Advisers (US) ed, to the Congress Council of Economic Advisers, , 2010

77. 193 J Apparatus, and method for examining pulmonary function using face image, and computer readable recording medium for recoring the same, Hong, K. Park, KR Patent, No. 1019789000000, , 2019

78. Oxygen saturation measuring method using image and computer readable storage medium of recording oxygen saturation measuring method using image, Hong, K., KR Patent, No. 1017525600000, , 2017

79. Theoretical developments and clinical experiments of measuring blood flow volume (BFV) at arteriovenous fistula (AVF) using a photoplethysmography (PPG) sensor, Chiang, PeiYu et al, Microsystem Technologies, 24,, 45874603, , 2018

80. Cuffless blood pressure prediction from ecg and ppg signals using Fourier transformation and amplitude randomization preprocessing for context aggregation network training, Treebupachatsakul, T. et al, 12.3, 159, , 2022

81. Method and estimating hemoglobin concentration using skin image, or health information and body information, and hemoglobin concentration estimating device performing method, Park, J., Hong, K., No. 1020220149107, , 2021