Dataset Information

Validation and Recalibration of Two Multivariable Prognostic Models for Survival and Independence in Acute Stroke.

ABSTRACT:

Introduction

Various prognostic models have been developed for acute stroke, including one based on age and five binary variables ('six simple variables' model; SSVMod) and one based on age plus scores on the National Institutes of Health Stroke Scale (NIHSSMod). The aims of this study were to externally validate and recalibrate these models, and to compare their predictive ability in relation to both survival and independence.

Methods

Data from a large clinical trial of oxygen therapy (n = 8003) were used to determine the discrimination and calibration of the models, using C-statistics, calibration plots, and Hosmer-Lemeshow statistics. Methods of recalibration in the large and logistic recalibration were used to update the models.

Results

For discrimination, both models functioned better for survival (C-statistics between .802 and .837) than for independence (C-statistics between .725 and .735). Both models showed slight shortcomings with regard to calibration, over-predicting survival and under-predicting independence; the NIHSSMod performed slightly better than the SSVMod. For the most part, there were only minor differences between ischaemic and haemorrhagic strokes. Logistic recalibration successfully updated the models for a clinical trial population.

Conclusions

Both prognostic models performed well overall in a clinical trial population. The choice between them is probably better based on clinical and practical considerations than on statistical considerations.

SUBMITTER: Sim J

PROVIDER: S-EPMC4881958 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:ObjectiveThe objective of this study was to develop valid prognostic models to predict mortality, dependency, and "death or dependency" for use in newly diagnosed Parkinson's disease (PD).MethodsThe models were developed in the Parkinsonism Incidence in North-East Scotland study (UK, 198 patients) and validated in the ParkWest study (Norway, 192 patients), cohorts that attempted to identify and follow-up all new PD cases in the study area. Dependency was defined using the Schwab & England scale. We selected variables measured at time of diagnosis to include in the models. Internal validation and external validation were performed by calculating C-statistics (discrimination) and plotting observed versus predicted risk in quantiles of predicted risk (calibration).ResultsOlder age, male sex, increased severity of axial features, and Charlson comorbidity index were independent prognostic factors in the mortality model. Increasing age, higher smoking history, increased severity of axial features, and lower MMSE score were independent predictors in the models of dependency and "death or dependency." Each model had very good internal calibration and very good or good discrimination (internal and external C-statistics for the models were 0.73-0.75 and 0.68-0.78, respectively). Although each model clearly separated patients into groups according to risk, they tended to overestimate risk in ParkWest. The models were recalibrated to the baseline risk in the ParkWest study and then calibrated well in this cohort.ConclusionsWe have developed prognostic models for predicting medium-term risk of important clinical outcomes in newly diagnosed PD. These models have validity for use for stratification of randomization, confounder adjustment, and case-mix correction, but they are inadequate for individualized prognostication. © 2017. The Authors. Movement Disorders published by Wiley Periodicals, Inc. on behalf of International Parkinson and Movement Disorder Society.

Project description:Background and Objectives: Acute kidney injury (AKI) that results from ischemia is a common clinical syndrome and correlates with high morbidity and mortality among hospitalized patients. However, a clinical tool to predict mortality risk of ischemic AKI is not available. In this study, we aimed to develop and validate models to predict the 30-day and 1-year mortality risk of hospitalized patients with ischemic AKI. Methods A total of 1,836 admissions with ischemic AKI were recruited from 277,898 inpatients admitted to three affiliated tertiary general hospitals of Central South University in China between January 2015 and December 2015. Patients in the final analysis were followed up for 1 year. Study patients were randomly divided in a 7:3 ratio to form the training cohort and validation cohort. Multivariable regression analyses were used for developing mortality prediction models. Results Hepatorenal syndrome, shock, central nervous system failure, Charlson comorbidity index (≥2 points), mechanical ventilation, renal function at discharge were independent risk factors for 30-day mortality after ischemic AKI, while malignancy, sepsis, heart failure, liver failure, Charlson comorbidity index (≥2 points), mechanical ventilation, and renal function at discharge were predictors for 1-year mortality. The area under the receiver operating characteristic curves (AUROCs) of 30-day prediction model were 0.878 (95% confidence interval (CI): 0.849-0.908) in the training cohort and 0.867 (95% CI: 0.820–0.913) in the validation cohort. The AUROCs of the 1-year mortality prediction in the training and validation cohort were 0.803 (95% CI: 0.772–0.834) and 0.788 (95% CI: 0.741–0.835), respectively. Conclusion Our easily applied prediction models can effectively identify individuals at high mortality risk within 30 days or 1 year in hospitalized patients with ischemic AKI. It can guide the optimal clinical management to minimize mortality after an episode of ischemic AKI.

Project description:Background and objectivesVery poor outcome despite intravenous thrombolysis (IVT) and mechanical thrombectomy (MT) occurs in about 1 of 4 patients with ischemic stroke and is associated with a high logistic and economic burden. We aimed to develop and validate a multivariable prognostic model to identify futile recanalization therapies (FRT) in patients undergoing those therapies.Materials and methodsPatients from a prospectively collected observational registry of a single academic stroke center treated with MT and/or IVT were included. The dataset was split into a training (N=1808, 80%) and internal validation (N=453, 20%) cohort. We used gradient boosted decision tree machine-learning models after k-NN imputation of 32 variables available at admission to predict FRT defined as modified Rankin-Scale (mRS) 5-6 at 3 months. We report feature importance, ability for discrimination, calibration and decision curve analysis.Results2261 patients with a median (IQR) age 75 years (64-83), 46% female, median NIHSS 9 (4-17), 34% IVT alone, 41% MT alone, 25% bridging were included. Overall 539 (24%) had FRT, more often in MT alone (34%) as compared to IVT alone (11%). Feature importance identified clinical variables (stroke severity, age, active cancer, prestroke disability), laboratory values (glucose, CRP, creatinine), imaging biomarkers (white matter hyperintensities) and onset-to-admission time as the most important predictors. The final model was discriminatory for predicting 3-month FRT (AUC 0.87, 95% CI 0.87-0.88) and had good calibration (Brier 0.12, 0.11-0.12). Overall performance was moderate (F1-score 0.63 ± 0.004) and decision curve analyses suggested higher mean net benefit at lower thresholds of treatment (up to 0.8).ConclusionsThis FRT prediction model can help inform shared decision making and identify the most relevant features in the emergency setting. While it might be particularly useful in low resource healthcare settings, incorporation of further multifaceted variables is necessary to further increase the predictive performance.

Project description:BackgroundBefore considering whether to use a multivariable (diagnostic or prognostic) prediction model, it is essential that its performance be evaluated in data that were not used to develop the model (referred to as external validation). We critically appraised the methodological conduct and reporting of external validation studies of multivariable prediction models.MethodsWe conducted a systematic review of articles describing some form of external validation of one or more multivariable prediction models indexed in PubMed core clinical journals published in 2010. Study data were extracted in duplicate on design, sample size, handling of missing data, reference to the original study developing the prediction models and predictive performance measures.Results11,826 articles were identified and 78 were included for full review, which described the evaluation of 120 prediction models. in participant data that were not used to develop the model. Thirty-three articles described both the development of a prediction model and an evaluation of its performance on a separate dataset, and 45 articles described only the evaluation of an existing published prediction model on another dataset. Fifty-seven percent of the prediction models were presented and evaluated as simplified scoring systems. Sixteen percent of articles failed to report the number of outcome events in the validation datasets. Fifty-four percent of studies made no explicit mention of missing data. Sixty-seven percent did not report evaluating model calibration whilst most studies evaluated model discrimination. It was often unclear whether the reported performance measures were for the full regression model or for the simplified models.ConclusionsThe vast majority of studies describing some form of external validation of a multivariable prediction model were poorly reported with key details frequently not presented. The validation studies were characterised by poor design, inappropriate handling and acknowledgement of missing data and one of the most key performance measures of prediction models i.e. calibration often omitted from the publication. It may therefore not be surprising that an overwhelming majority of developed prediction models are not used in practice, when there is a dearth of well-conducted and clearly reported (external validation) studies describing their performance on independent participant data.

Project description:Risk calculators can facilitate shared medical decision-making 1 . Demographics, comorbidities, medication use, geographic region, and other factors may increase the risk for COVID-19-related complications among patients with IBD 2,3 . Develop an individualized prognostic risk prediction tool for predicting the probability of adverse COVID-19 outcomes in patients with IBD. This study developed and validated prognostic penalized logistic regression models 4 using reports to Surveillance Epidemiology of Coronavirus Under Research Exclusion for Inflammatory Bowel Disease (SECURE-IBD) from March-October 2020. Model development was done using a training data set (85% of cases reported March 13 - September 15, 2020), and model validation was conducted using a test data set (the remaining 15% of cases plus all cases reported September 16-October 20, 2020). COVID-19 related:Hospitalization+: composite outcome of hospitalization, ICU admission, mechanical ventilation, or deathICU+: composite outcome of ICU admission, mechanical ventilation, or deathDeathWe assessed the resulting models' discrimination using the area under the curve (AUC) of the receiver-operator characteristic (ROC) curves and reported the corresponding 95% confidence intervals (CIs). We included 2709 cases from 59 countries (mean age 41.2 years [s.d. 18], 50.2% male). A total of 633 (24%) were hospitalized, 137 (5%) were admitted to the ICU or intubated, and 69 (3%) died. 2009 patients comprised the training set and 700 the test set.The models demonstrated excellent discrimination, with a test set AUC (95% CI) of 0.79 (0.75, 0.83) for Hospitalization+, 0.88 (0.82, 0.95) for ICU+, and 0.94 (0.89, 0.99) for Death. Age, comorbidities, corticosteroid use, and male gender were associated with higher risk of death, while use of biologic therapies was associated with a lower risk. Prognostic models can effectively predict who is at higher risk for COVID-19-related adverse outcomes in a population of IBD patients. A free online risk calculator ( https://covidibd.org/covid-19-risk-calculator/ ) is available for healthcare providers to facilitate discussion of risks due to COVID-19 with IBD patients. The tool numerically and visually summarizes the patient's probabilities of adverse outcomes and associated CIs. Helping physicians identify their highest-risk patients will be important in the coming months as cases rise in the US and worldwide. This tool can also serve as a model for risk stratification in other chronic diseases. Question: How well can a multivariable risk model predict the risk of hospitalization, intensive care unit (ICU) stay, or death due to COVID-19 in patients with inflammatory bowel disease (IBD)?Findings: Multivariable prediction models developed using data from an international voluntary registry of IBD patients and available for use online ( https://covidibd.org/ ) have very good discrimination for predicting hospitalization (Test set AUC 0.79) and excellent discrimination for ICU admission (Test set AUC 0.88) and death (Test set AUC 0.94). The models were developed with a training sample of 2009 cases and validated in an independent test sample of 700 cases comprised of a random sub-sample of cases and all cases entered in the registry during a one-month period after model development. Meaning: This risk prediction model may serve as an effective tool for healthcare providers to facilitate conversations about COVID-19-related risks with IBD patients.