Detection of calibration drift in clinical prediction models to inform model updating.
Ontology highlight
ABSTRACT:
SUBMITTER: Davis SE
PROVIDER: S-EPMC8627243 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:ObjectiveClinical prediction models require updating as performance deteriorates over time. We developed a testing procedure to select updating methods that minimizes overfitting, incorporates uncertainty associated with updating sample sizes, and is applicable to both parametric and nonparametric models.Materials and methodsWe describe a procedure to select an updating method for dichotomous outcome models by balancing simplicity against accuracy. We illustrate the test's properties on simulated scenarios of population shift and 2 models based on Department of Veterans Affairs inpatient admissions.ResultsIn simulations, the test generally recommended no update under no population shift, no update or modest recalibration under case mix shifts, intercept correction under changing outcome rates, and refitting under shifted predictor-outcome associations. The recommended updates provided superior or similar calibration to that achieved with more complex updating. In the case study, however, small update sets lead the test to recommend simpler updates than may have been ideal based on subsequent performance.DiscussionOur test's recommendations highlighted the benefits of simple updating as opposed to systematic refitting in response to performance drift. The complexity of recommended updating methods reflected sample size and magnitude of performance drift, as anticipated. The case study highlights the conservative nature of our test.ConclusionsThis new test supports data-driven updating of models developed with both biostatistical and machine learning approaches, promoting the transportability and maintenance of a wide array of clinical prediction models and, in turn, a variety of applications relying on modern prediction tools.
| S-EPMC6857513 | biostudies-literature
Project description:There is growing interest in developing clinical prediction models (CPMs) to aid local healthcare decision-making. Frequently, these CPMs are developed in isolation across different populations, with repetitive de novo derivation a common modelling strategy. However, this fails to utilise all available information and does not respond to changes in health processes through time and space. Alternatively, model updating techniques have previously been proposed that adjust an existing CPM to suit the new population, but these techniques are restricted to a single model. Therefore, we aimed to develop a generalised method for updating and aggregating multiple CPMs. The proposed "hybrid method" re-calibrates multiple CPMs using stacked regression while concurrently revising specific covariates using individual participant data (IPD) under a penalised likelihood. The performance of the hybrid method was compared with existing methods in a clinical example of mortality risk prediction after transcatheter aortic valve implantation, and in 2 simulation studies. The simulation studies explored the effect of sample size and between-population-heterogeneity on the method, with each representing a situation of having multiple distinct CPMs and 1 set of IPD. When the sample size of the IPD was small, stacked regression and the hybrid method had comparable but highest performance across modelling methods. Conversely, in large IPD samples, development of a new model and the hybrid method gave the highest performance. Hence, the proposed strategy can inform the choice between utilising existing CPMs or developing a model de novo, thereby incorporating IPD, existing research, and prior (clinical) knowledge into the modelling strategy.
| S-EPMC5873448 | biostudies-literature
Project description:Our primary objective is to provide the clinical informatics community with an introductory tutorial on calibration measurements and calibration models for predictive models using existing R packages and custom implemented code in R on real and simulated data. Clinical predictive model performance is commonly published based on discrimination measures, but use of models for individualized predictions requires adequate model calibration. This tutorial is intended for clinical researchers who want to evaluate predictive models in terms of their applicability to a particular population. It is also for informaticians and for software engineers who want to understand the role that calibration plays in the evaluation of a clinical predictive model, and to provide them with a solid starting point to consider incorporating calibration evaluation and calibration models in their work. Covered topics include (1) an introduction to the importance of calibration in the clinical setting, (2) an illustration of the distinct roles that discrimination and calibration play in the assessment of clinical predictive models, (3) a tutorial and demonstration of selected calibration measurements, (4) a tutorial and demonstration of selected calibration models, and (5) a brief discussion of limitations of these methods and practical suggestions on how to use them in practice.
| S-EPMC7075534 | biostudies-literature
Project description:BackgroundPrediction models inform many medical decisions, but their performance often deteriorates over time. Several discrete-time update strategies have been proposed in the literature, including model recalibration and revision. However, these strategies have not been compared in the dynamic updating setting.MethodsWe used post-lung transplant survival data during 2010-2015 and compared the Brier Score (BS), discrimination, and calibration of the following update strategies: (1) never update, (2) update using the closed testing procedure proposed in the literature, (3) always recalibrate the intercept, (4) always recalibrate the intercept and slope, and (5) always refit/revise the model. In each case, we explored update intervals of every 1, 2, 4, and 8 quarters. We also examined how the performance of the update strategies changed as the amount of old data included in the update (i.e., sliding window length) increased.ResultsAll methods of updating the model led to meaningful improvement in BS relative to never updating. More frequent updating yielded better BS, discrimination, and calibration, regardless of update strategy. Recalibration strategies led to more consistent improvements and less variability over time compared to the other updating strategies. Using longer sliding windows did not substantially impact the recalibration strategies, but did improve the discrimination and calibration of the closed testing procedure and model revision strategies.ConclusionsModel updating leads to improved BS, with more frequent updating performing better than less frequent updating. Model recalibration strategies appeared to be the least sensitive to the update interval and sliding window length.
| S-EPMC8647501 | biostudies-literature
Project description:Objective:Predictive analytics create opportunities to incorporate personalized risk estimates into clinical decision support. Models must be well calibrated to support decision-making, yet calibration deteriorates over time. This study explored the influence of modeling methods on performance drift and connected observed drift with data shifts in the patient population. Materials and Methods:Using 2003 admissions to Department of Veterans Affairs hospitals nationwide, we developed 7 parallel models for hospital-acquired acute kidney injury using common regression and machine learning methods, validating each over 9 subsequent years. Results:Discrimination was maintained for all models. Calibration declined as all models increasingly overpredicted risk. However, the random forest and neural network models maintained calibration across ranges of probability, capturing more admissions than did the regression models. The magnitude of overprediction increased over time for the regression models while remaining stable and small for the machine learning models. Changes in the rate of acute kidney injury were strongly linked to increasing overprediction, while changes in predictor-outcome associations corresponded with diverging patterns of calibration drift across methods. Conclusions:Efficient and effective updating protocols will be essential for maintaining accuracy of, user confidence in, and safety of personalized risk predictions to support decision-making. Model updating protocols should be tailored to account for variations in calibration drift across methods and respond to periods of rapid performance drift rather than be limited to regularly scheduled annual or biannual intervals.
| S-EPMC6080675 | biostudies-literature
Project description:BackgroundClinical prediction models are often not evaluated properly in specific settings or updated, for instance, with information from new markers. These key steps are needed such that models are fit for purpose and remain relevant in the long-term. We aimed to present an overview of methodological guidance for the evaluation (i.e., validation and impact assessment) and updating of clinical prediction models.MethodsWe systematically searched nine databases from January 2000 to January 2022 for articles in English with methodological recommendations for the post-derivation stages of interest. Qualitative analysis was used to summarize the 70 selected guidance papers.ResultsKey aspects for validation are the assessment of statistical performance using measures for discrimination (e.g., C-statistic) and calibration (e.g., calibration-in-the-large and calibration slope). For assessing impact or usefulness in clinical decision-making, recent papers advise using decision-analytic measures (e.g., the Net Benefit) over simplistic classification measures that ignore clinical consequences (e.g., accuracy, overall Net Reclassification Index). Commonly recommended methods for model updating are recalibration (i.e., adjustment of intercept or baseline hazard and/or slope), revision (i.e., re-estimation of individual predictor effects), and extension (i.e., addition of new markers). Additional methodological guidance is needed for newer types of updating (e.g., meta-model and dynamic updating) and machine learning-based models.ConclusionSubstantial guidance was found for model evaluation and more conventional updating of regression-based models. An important development in model evaluation is the introduction of a decision-analytic framework for assessing clinical usefulness. Consensus is emerging on methods for model updating.
| S-EPMC9742671 | biostudies-literature
Project description:Rationale & objectiveThere is limited evidence to guide follow-up after acute kidney injury (AKI). Knowledge gaps include which patients to prioritize, at what time point, and for mitigation of which outcomes. In this study, we sought to compare the net benefit of risk model-based clinical decisions following AKI.Study designExternal validation of 2 risk models of AKI outcomes: the Grampian -Aberdeen (United Kingdom) AKI readmissions model and the Alberta (Canada) kidney disease risk model of chronic kidney disease (CKD) glomerular (G) filtration rate categories 4 and 5 (CKD G4 and G5). Process mining to delineate existing care pathways.Setting & participantsValidation was based on data from adult hospital survivors of AKI from Grampian, 2011-2013.PredictorsKDIGO-based measures of AKI severity and comorbidities specified in the original models.OutcomesDeath or readmission within 90 days for all hospital survivors. Progression to new CKD G4-G5 for patients surviving at least 90 days after AKI.Analytical approachDecision curve analysis to assess the "net benefit" of use of risk models to guide clinical care compared to alternative approaches (eg, prioritizing all AKI, severe AKI, or only those without kidney recovery).Results26,575 of 105,461 hospital survivors in Grampian (mean age, 60.9 ± 19.8 [SD] years) were included for validation of the death or readmission model, and 9,382 patients (mean age, 60.9 ± 19.8 years) for the CKD G4-G5 model. Both models discriminated well (area under the curve [AUC], 0.77 and 0.86, respectively). Decision curve analysis showed greater net benefit for follow up of all AKI than only severe AKI in most cases. Both original and refitted models provided net benefit superior to any other decision strategy. In process mining of all hospital discharges, 41% of readmissions and deaths occurred among people recovering after AKI. 1,464 of 3,776 people (39%) readmitted after AKI had received no intervening monitoring.LimitationsBoth original models overstated risks, indicating a need for regular updating.ConclusionsFollow up after AKI has potential net benefit for preempting readmissions, death, and subsequent CKD progression. Decisions could be improved by using risk models and by focusing on AKI across a full spectrum of severity. The current lack of monitoring among many with poor outcomes indicates possible opportunities for implementation of decision support.
| S-EPMC8234511 | biostudies-literature
Project description:ObjectiveTo illustrate the problem of subpopulation miscalibration, to adapt an algorithm for recalibration of the predictions, and to validate its performance.Materials and methodsIn this retrospective cohort study, we evaluated the calibration of predictions based on the Pooled Cohort Equations (PCE) and the fracture risk assessment tool (FRAX) in the overall population and in subpopulations defined by the intersection of age, sex, ethnicity, socioeconomic status, and immigration history. We next applied the recalibration algorithm and assessed the change in calibration metrics, including calibration-in-the-large.Results1 021 041 patients were included in the PCE population, and 1 116 324 patients were included in the FRAX population. Baseline overall model calibration of the 2 tested models was good, but calibration in a substantial portion of the subpopulations was poor. After applying the algorithm, subpopulation calibration statistics were greatly improved, with the variance of the calibration-in-the-large values across all subpopulations reduced by 98.8% and 94.3% in the PCE and FRAX models, respectively.DiscussionPrediction models in medicine are increasingly common. Calibration, the agreement between predicted and observed risks, is commonly poor for subpopulations that were underrepresented in the development set of the models, resulting in bias and reduced performance for these subpopulations. In this work, we empirically evaluated an adapted version of the fairness algorithm designed by Hebert-Johnson et al. (2017) and demonstrated its use in improving subpopulation miscalibration.ConclusionA postprocessing and model-independent fairness algorithm for recalibration of predictive models greatly decreases the bias of subpopulation miscalibration and thus increases fairness and equality.
| S-EPMC7936516 | biostudies-literature
Project description:Until recently, no prediction models for Lynch syndrome (LS) had been validated for PMS2 mutation carriers. We aimed to evaluate MMRpredict and PREMM5 in a clinical cohort and for PMS2 mutation carriers specifically. In a retrospective, clinic-based cohort we calculated predictions for LS according to MMRpredict and PREMM5. The area under the operator receiving characteristic curve (AUC) was compared between MMRpredict and PREMM5 for LS patients in general and for different LS genes specifically. Of 734 index patients, 83 (11%) were diagnosed with LS; 23 MLH1, 17 MSH2, 31 MSH6 and 12 PMS2 mutation carriers. Both prediction models performed well for MLH1 and MSH2 (AUC 0.80 and 0.83 for PREMM5 and 0.79 for MMRpredict) and fair for MSH6 mutation carriers (0.69 for PREMM5 and 0.66 for MMRpredict). MMRpredict performed fair for PMS2 mutation carriers (AUC 0.72), while PREMM5 failed to discriminate PMS2 mutation carriers from non-mutation carriers (AUC 0.51). The only statistically significant difference between PMS2 mutation carriers and non-mutation carriers was proximal location of colorectal cancer (77 vs. 28%, p?<?0.001). Adding location of colorectal cancer to PREMM5 considerably improved the models performance for PMS2 mutation carriers (AUC 0.77) and overall (AUC 0.81 vs. 0.72). We validated these results in an external cohort of 376 colorectal cancer patients, including 158 LS patients. MMRpredict and PREMM5 cannot adequately identify PMS2 mutation carriers. Adding location of colorectal cancer to PREMM5 may improve the performance of this model, which should be validated in larger cohorts.
| S-EPMC5999171 | biostudies-literature
Project description:Machine learning methods are widely used in drug discovery and toxicity prediction. While showing overall good performance in cross-validation studies, their predictive power (often) drops in cases where the query samples have drifted from the training data's descriptor space. Thus, the assumption for applying machine learning algorithms, that training and test data stem from the same distribution, might not always be fulfilled. In this work, conformal prediction is used to assess the calibration of the models. Deviations from the expected error may indicate that training and test data originate from different distributions. Exemplified on the Tox21 datasets, composed of chronologically released Tox21Train, Tox21Test and Tox21Score subsets, we observed that while internally valid models could be trained using cross-validation on Tox21Train, predictions on the external Tox21Score data resulted in higher error rates than expected. To improve the prediction on the external sets, a strategy exchanging the calibration set with more recent data, such as Tox21Test, has successfully been introduced. We conclude that conformal prediction can be used to diagnose data drifts and other issues related to model calibration. The proposed improvement strategy-exchanging the calibration data only-is convenient as it does not require retraining of the underlying model.
| S-EPMC8082859 | biostudies-literature