Evaluating AI-enabled EHR Quality : A 4I Framework and Expert MCDM Approach

다국어 초록 (Multilingual Abstract)

This dissertation builds a traceable design for evaluating AI-enabled electronic health record quality through a 4I framework and expert-based multi-criteria decision-making. The problem addressed is practical: EHR interface conditions, information quality, clinician-system use, and downstream consequences are often examined in separate streams, making it difficult to trace how system conditions are carried into clinical work. Part 1 provides the evidence base through a PRISMA-guided integrative review of literature published between 2005 and 2025. Seventy studies were synthesized into a 29-indicator dictionary organized under Interface, Information, Interaction, and Outcome. The 4I structure keeps the familiar socio-technical logic of interface, information, and interaction, while adding Outcome explicitly so that safety culture, burnout, error reduction, adoption, workflow standardization, and decision-making support remain visible.
Part 2 applies this evidence-derived dictionary in a multilingual, multi-institutional expert evaluation. The questionnaire was refined through a two-round Content Validity Index procedure. Round A served as a relationship-screening stage, reducing the number of influence judgments required before formal scoring. Round B collected expert scores for AHP, Entropy-TOPSIS, and bounded two-tier DEMATEL using Chinese, English, and Korean questionnaire versions. This staged design was selected for a high-burden clinical fieldwork setting: asking busy hospital professionals to complete a full 29 × 29 influence matrix, or a lengthy interview sequence, would have been difficult to sustain with acceptable response quality. The final Round B dataset contained 34 raw responses and 32 primary valid cases after cleaning. Respondents represented multiple practice countries or regions and 19 identifiable institution groups after conservative abbreviation harmonization. These descriptors are reported to show panel breadth, not to support country-, language-, or hospital-level inference. AHP estimated layer importance. Entropy-TOPSIS summarized an exploratory priority profile from the 29-item rating block. DEMATEL-2A examined five screened forward layer-level paths, and DEMATEL-2B examined 67 retained indicator-level relations from Round A. The statistical files were checked through local Python recomputation and file-manifest locking.
The findings indicate that AI-enabled EHR quality cannot be read from technical performance or interface usability alone. Expert judgments point to the need to examine information reliability, system usability, workflow fit, organizational support, and downstream consequences together. Making Outcome explicit added value by keeping adoption, safety, burnout, error reduction, and decision-making support inside the assessment structure rather than leaving them as background implications. The interpretation is deliberately bounded: DEMATEL is reported as a retained-structure reading, not as a full 29 × 29 causal network; the expert panel supports structured interpretation, not population-level inference; and country or language differences are descriptive only. Future work may extend the framework into scenario-based AIEHR simulation and real-world data analysis, including target-trial-style designs and heterogeneous-effect methods such as causal forests. Those extensions are reserved for later research and are not treated as completed empirical work in this dissertation.

국문 초록 (Abstract)

본 박사학위논문은 AI 기반 전자의무기록（EHR）의 품질을 인터페이스, 정보, 상호작용, 결과의 4I 구조 안에서 추적 가능하게 평가하기 위한 연구 설계를 제시한다. 본 연구의 출발점은 기존 EHR 및 의료정보시스템 연구가 사용성, 데이터 품질, 워크플로우 적합성, 신뢰, 안전성 등 중요한 문제를 보고해 왔음에도, 이러한 조건들이 임상 사용 경험과 후속 결과로 어떻게 연결되는지를 일관되게 설명하는 데 한계가 있다는 점이다. 이에 본 연구는 기존 3I 관점에 결과 층을 명시적으로 추가하여 4I 구조를 정립하고, 문헌에서 보고된 주요 요인을 29개 지표 체계로 정리하였다.
첫 번째 연구에서는 2005년부터 2025년까지의 문헌을 대상으로 PRISMA 지침 기반 통합문헌고찰을 수행하였으며, 최종 70편의 연구를 바탕으로 인터페이스, 정보, 상호작용, 결과에 걸친 29개 지표 사전을 도출하였다. 두 번째 연구에서는 이 문헌 기반 지표 체계를 다국어·다기관 전문가 평가에 적용하였다. 설문지는 내용타당도지수（Content Validity Index, CVI） 절차를 통해 관련성과 명확성을 점검하였고, Round A에서는 영향 관계를 선별하였으며, Round B에서는 AHP, Entropy-TOPSIS, bounded two-tier DEMATEL 분석을 수행하였다. 최종 Round B 자료는 34개의 원자료 응답과 32개의 주요 유효 사례로 구성되었으며, 중국어, 영어, 한국어 설문 버전을 통해 수집되었다.
분석 결과는 AI 기반 EHR 품질이 단순한 기술 성능이나 인터페이스 사용성만으로 설명될 수 없으며, 정보 신뢰성, 시스템 사용성, 워크플로우 적합성, 조직적 지원, 후속 결과를 함께 고려해야 함을 보여준다. 특히 결과 층을 명시적으로 포함함으로써 도입, 안전문화, 번아웃, 오류 감소, 의사결정 지원과 같은 결과 요인이 평가 구조 안에서 추적 가능하게 유지된다. 본 연구의 해석은 전문가 판단과 선별된 관계 구조에 기반한 것으로, 전체 29 × 29 인과망이나 모집단 수준의 통계적 추론을 주장하지 않는다. 따라서 본 논문은 AI 기반 EHR 평가를 위한 문헌 기반 4I 구조와 이를 활용한 제한적 전문가 다기준 의사결정（MCDM） 분석을 연결하는 추적 가능한 연구 틀을 제공한다.

목차 (Table of Contents)

• I. Introduction 1
• 1.1 Background and research problem 1
• 1.2 Research gap and the need for an integrated evaluation structure 4
• 1.3 Research aim and objectives 5
• 1.4 Research questions 6
• 1.5 Dissertation structure 7
• II. Literature Review and Conceptual Framework 9
• 2.1 From traditional EHR problems to AI-enabled EHR evaluation 9
• 2.2 Conceptual Basis of the 4I structure 11
• 2.3 Terminology standardization used in this dissertation 12
• 2.4 Classification principle and coding guide 14
• 2.5 Mechanism pathway across the 4I layers 16
• 2.6 AI-specific scope and boundaries of interpretation 17
• 2.7 Practical relevance for hospital decision-making 18
• III. Overall Dissertation Design 19
• 3.1 Completed dissertation scope and broader research program 19
• 3.2 Role of Part 1 in the dissertation 20
• 3.3 Completed Part 2 empirical design and analysis 21
• 3.4 Future extension: AIEHR simulation and real-world data analysis 23
• IV. Part 1 Methods: PRISMA-guided Integrative Review . 24
• 4.1 Research design 24
• 4.2 Literature identification and selection 25
• 4.3 Data extraction and labelling procedures 29
• 4.4 Quality assessment 33
• 4.5 Derivation and operationalization of the 29 indicators 34
• 4.6 Justification for inclusion of the 29 indicators 35
• V. Part 1 Results: The 4I Evidence Base 37
• 5.1 Overview of the 4I evidence structure 37
• 5.2 Multi-layered variable structure across the 4I layers 46
• 5.3 Drivers, barriers, and mixed conditions for adoption and sustained use 52
• 5.4 Recurring cross-layer linkages described in the evidence base 54
• 5.5 Summary of Part 1 contributions 58
• VI. Part 1 Discussion and Dissertation Implications 60
• 6.1 From interface modification to results-oriented redesign 60
• 6.2 Value of a hierarchical and traceable representation 61
• 6.3 Implications for governance and staged adoption 62
• 6.4 Limitations and boundary conditions 64
• 6.5 Contribution to the overall dissertation 67
• VII. Part 2 Methods: Expert Validation, Round A Screening, and Rechecked MCDM Design 69
• 7.1 Research design and empirical sequence 69
• 7.1.1 Rationale for using MCDM and bounded DEMATEL approach in a clinical expert setting 70
• 7.2 Expert panel, multilingual administration, and institutional context 73
• 7.3 Content Validity Index procedure 75
• 7.4 CVI results and instrument refinement 77
• 7.5 Round A structure-screening design 79
• 7.6 Round A data processing and retained-edge freeze 81
• 7.7 Round B formal scoring instrument 83
• 7.8 Reproducible calculation workflow and final statistical lock 85
• 7.9 MCDM analysis plan 87
• 7.10 Claim boundaries for Part 2 interpretation 90
• VIII. Part 2 Results: Round A Screening and Final Rechecked Round B Outputs 92
• 8.1 Data status and reproducibility lock 92
• 8.2 Round A audited panel and retained structure 93
• 8.3 Round B module completeness, multilingual administration, and institutional profile 96
• 8.4 AHP layer-weight results 99
• 8.5 Rating descriptives and AHP-adjusted gaps 100
• 8.6 Entropy-TOPSIS exploratory priority profile 103
• 8.7 DEMATEL-2A layer-level structure 105
• 8.8 DEMATEL-2B retained-block structure 106
• 8.9 Robustness and reporting boundary checks 108
• 8.10 Research-question mapping after Python rechecks 110
• IX. Discussion: Part 2 Findings, Contributions, and Future Research Direction 112
• 9.1 Integrated reading of the completed dissertation 112
• 9.2 Interpretation of the Part 2 findings 114
• 9.3 Why the Outcome layer remains necessary 115
• 9.4 Part 2 contribution, contextual breadth, and methodological boundaries 117
• 9.5 Future extension: AIEHR simulation and real-world analysis 121
• X. Conclusion 125
• 10.1 Summary of the completed dissertation 125
• 10.2 Main contributions 125
• 10.3 Limitations 126
• 10.4 Future research 127
• 10.5 Closing statement 128
• References 129