How to Develop, Validate, and Compare Clinical Prediction Models Involving Radiological Parameters: Study Design and Statistical Methods

다국어 초록 (Multilingual Abstract)

Clinical prediction models are developed to calculate estimates of the probability of the presence/occurrence or future course of a particular prognostic or diagnostic outcome from multiple clinical or non-clinical parameters. Radiologic imaging techniques are being developed for accurate detection and early diagnosis of disease, which will eventually affect patient outcomes. Hence, results obtained by radiological means, especially diagnostic imaging, are frequently incorporated into a clinical prediction model as important predictive parameters, and the performance of the prediction model may improve in both diagnostic and prognostic settings. This article explains in a conceptual manner the overall process of developing and validating a clinical prediction model involving radiological parameters in relation to the study design and statistical methods. Collection of a raw dataset; selection of an appropriate statistical model; predictor selection; evaluation of model performance using a calibration plot, Hosmer-Lemeshow test and c-index; internal and external validation; comparison of different models using c-index, net reclassification improvement, and integrated discrimination improvement; and a method to create an easy-to-use prediction score system will be addressed. This article may serve as a practical methodological reference for clinical researchers.

참고문헌 (Reference)

1 Rufibach K, "Use of Brier score to assess binary predictions" 63 : 938-939, 2010

2 Cook NR, "Use and misuse of the receiver operating characteristic curve in risk prediction" 115 : 928-935, 2007

3 Janssen KJ, "Updating methods improved the performance of a clinical prediction model in new patients" 61 : 76-86, 2008

4 Collins GS, "Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement" 162 : 55-63, 2015

5 Moons KG, "Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration" 162 : w1-w73, 2015

6 Ware JH, "The limitations of risk factors as prognostic tools" 355 : 2615-2617, 2006

7 Pepe MS, "Testing for improvement in prediction model performance" 32 : 1467-1482, 2013

8 Sunshine JH, "Technology assessment for radiologists" 230 : 309-314, 2004

9 Vergouwe Y, "Substantial effective sample sizes were required for external validation studies of predictive logistic regression models" 58 : 475-483, 2005

10 Little RJA, "Statistical analysis with missing data" John Wiley & Sons 2014

11 Collins GS, "Sample size considerations for the external validation of a multivariable prognostic model: a resampling study" 35 : 214-226, 2016

12 Bossuyt PM, "STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies" 277 : 826-832, 2015

13 Yang HI, "Risk estimation for hepatocellular carcinoma in chronic hepatitis B (REACH-B): development and validation of a predictive score" 12 : 568-574, 2011

14 Vittinghoff E, "Relaxing the rule of ten events per variable in logistic and Cox regression" 165 : 710-718, 2007

15 Harrell FE, "Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis" Springer 2001

16 박성호, "Receiver Operating Characteristic (ROC) Curve: Practical Review for Radiologists" 대한영상의학회 5 (5): 11-18, 2004

17 Pencina MJ, "Quantifying discrimination of Framingham risk functions with different survival C statistics" 31 : 1543-1553, 2012

18 Suh YJ, "Prognostic value of SYNTAX score based on coronary computed tomography angiography" 199 : 460-466, 2015

19 Pepe MS, "Problems with risk reclassification methods for evaluating prediction models" 173 : 1327-1335, 2011

20 Sullivan LM, "Presentation of multivariate data for clinical use: The Framingham Study risk score functions" 23 : 1631-1660, 2004

21 Lee K, "Predictors of recurrent stroke in patients with ischemic stroke: comparison study between transesophageal echocardiography and cardiac CT" 276 : 381-389, 2015

22 Pepe MS, "Net risk reclassification p values: valid or misleading?" 106 : dju041-, 2014

23 Leening MJ, "Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician’s guide" 160 : 122-131, 2014

24 Demler OV, "Misuse of DeLong test to compare AUCs for nested models" 31 : 2577-2587, 2012

25 Shaw LJ, "Long-term prognosis after coronary artery calcification testing in asymptomatic patients: a cohort study" 163 : 14-21, 2015

26 Pepe MS, "Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker" 159 : 882-890, 2004

27 Pencina MJ, "Interpreting incremental value of markers added to risk prediction models" 176 : 473-481, 2012

28 Steyerberg EW, "Internal validation of predictive models: efficiency of some procedures for logistic regression analysis" 54 : 774-781, 2001

29 Widera C, "Incremental prognostic value of biomarkers beyond the GRACE (Global Registry of Acute Coronary Events) score and high-sensitivity cardiac troponin T in non-ST-elevation acute coronary syndrome" 59 : 1497-1505, 2013

30 Peduzzi P, "Importance of events per independent variable in proportional hazards regression analysis. II. Accuracy and precision of regression estimates" 48 : 1503-1510, 1995

31 곽진영, "Image Reporting and Characterization System for Ultrasound Features of Thyroid Nodules: Multicentric Korean Retrospective Study" 대한영상의학회 14 (14): 110-117, 2013

32 D’Agostino R, "Handbook of statistics: advances in survival analysis. Vol 23" Elsevier 1-25, 2004

33 D’Agostino RB Sr, "General cardiovascular risk profile for use in primary care: the Framingham Heart Study" 117 : 743-753, 2008

34 Collins GS, "External validation of multivariable prediction models: a systematic review of methodological conduct and reporting" 14 : 40-, 2014

35 Pencina MJ, "Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers" 30 : 11-21, 2011

36 Van Calster B, "Extending the c-statistic to nominal polytomous outcomes: the Polytomous Discrimination Index" 31 : 2610-2626, 2012

37 Kerr KF, "Evaluating the incremental value of new biomarkers with integrated discrimination improvement" 174 : 364-374, 2011

38 Pencina MJ, "Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond" 27 : 157-172, 2008

39 Royston P, "Dichotomizing continuous predictors in multiple regression: a bad idea" 25 : 127-141, 2006

40 Schnabel RB, "Development of a risk score for atrial fibrillation (Framingham Heart Study): a community-based cohort study" 373 : 739-745, 2009

41 Imperiale TF, "Derivation and Validation of a Scoring System to Stratify Risk for Advanced Colorectal Neoplasia in Asymptomatic Adults: A Cross-sectional Study" 163 : 339-346, 2015

42 윤연이, "Current Roles and Future Applications of Cardiac CT: Risk Stratification of Coronary Artery Disease" 대한영상의학회 15 (15): 4-11, 2014

43 Kim SY, "Coronary computed tomography angiography for selecting coronary artery bypass graft surgery candidates" 95 : 1340-1346, 2013

44 Wolbers M, "Concordance for prognostic models with competing risks" 15 : 526-539, 2014

45 DeLong ER, "Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach" 44 : 837-845, 1988

46 Pepe MS, "Commentary: reporting standards are needed for evaluations of risk reclassification" 40 : 1106-1108, 2011

47 Tjur T, "Coefficients of determination in logistic regression models—A new proposal: the coefficient of discrimination" 63 : 366-372, 2009

48 Bossuyt PM, "Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 0.4" The Cochrane Collaboration 2008

49 Steyerberg EW, "Clinical prediction models: a practical approach to development, validation, and updating" Springer 2009

50 Austin PC, "Bootstrap model selection had similar performance for selecting authentic and noise variables compared to backward variable elimination: a simulation study" 61 : 1009-1017, 2008

51 Austin PC, "Bootstrap methods for developing predictive models" 58 : 131-137, 2004

52 Austin PC, "Automated variable selection methods for logistic regression produced unstable models for predicting acute myocardial infarction mortality" 57 : 1138-1146, 2004

53 Hosmer Jr DW, "Applied logistic regression" John Wiley & Sons 2004

54 Van Oirbeek R, "An application of Harrell’s C-index to PH frailty models" 29 : 3160-3171, 2010

55 Altman DG, "Absence of evidence is not evidence of absence" 311 : 485-, 1995

56 Peduzzi P, "A simulation study of the number of events per variable in logistic regression analysis" 49 : 1373-1379, 1996

57 Nagelkerke NJ, "A note on a general definition of the coefficient of determination" 78 : 691-692, 1991

58 Hosmer DW, "A comparison of goodness-of-fit tests for the logistic regression model" 16 : 965-980, 1997