Dataset Information

Development and validation of multivariable mortality risk-prediction models in older people undergoing an interRAI home-care assessment (RiskOP)

ABSTRACT:

Background

Currently, one-year survival of older people with complex co-morbidities is unpredictable. Identifying older adults with a reduced life expectancy will lead to more targeted care and better healthcare resource allocation.

Methods

Development and validation of one-year and three-month mortality risks in people aged ?65 years who had completed an International Resident Assessment Instrument-Home Care (interRAI-HC) assessment between July 2012 and March 2018. Data was split into development (90%) and validation data sets (10%). A multivariable logistic regression model using data from 108 interRAI questions across multiple domains was developed and validated using discrimination metrics and calibration curves. Variables each explaining at least 1% of the model were then used to develop and validate a parsimonious model. Subgroups by sex, age, ethnicity, and comorbidities were evaluated.

Findings

There were 104,436 persons (60.2% female; mean age 82.1 years) in the study cohort of whom 20,972 (20.1%) died within one year. The full multivariable model had area under the curves (AUCs) of 0.778 to 0.795 in the 5 validation datasets and was well calibrated. After variable reduction a parsimonious model consisted of 16 variables and was well calibrated and the AUC remained high: 0.773 (0.769 to 0.777). The three-month parsimonious model comprised 22 variables and was well calibrated with an AUC of 0.843 (95%CI: 0.839 to 0.848).

Interpretation

These community-based risk prediction models accurately predict mortality in older people with complex co-morbidities. They may contribute to both forecasting for policy making and clinical decision making regarding an individual's needs.

Funding

The New Zealand Health Research Council.

SUBMITTER: Pickering J

PROVIDER: S-EPMC7788437 | biostudies-literature | 2020 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundBefore considering whether to use a multivariable (diagnostic or prognostic) prediction model, it is essential that its performance be evaluated in data that were not used to develop the model (referred to as external validation). We critically appraised the methodological conduct and reporting of external validation studies of multivariable prediction models.MethodsWe conducted a systematic review of articles describing some form of external validation of one or more multivariable prediction models indexed in PubMed core clinical journals published in 2010. Study data were extracted in duplicate on design, sample size, handling of missing data, reference to the original study developing the prediction models and predictive performance measures.Results11,826 articles were identified and 78 were included for full review, which described the evaluation of 120 prediction models. in participant data that were not used to develop the model. Thirty-three articles described both the development of a prediction model and an evaluation of its performance on a separate dataset, and 45 articles described only the evaluation of an existing published prediction model on another dataset. Fifty-seven percent of the prediction models were presented and evaluated as simplified scoring systems. Sixteen percent of articles failed to report the number of outcome events in the validation datasets. Fifty-four percent of studies made no explicit mention of missing data. Sixty-seven percent did not report evaluating model calibration whilst most studies evaluated model discrimination. It was often unclear whether the reported performance measures were for the full regression model or for the simplified models.ConclusionsThe vast majority of studies describing some form of external validation of a multivariable prediction model were poorly reported with key details frequently not presented. The validation studies were characterised by poor design, inappropriate handling and acknowledgement of missing data and one of the most key performance measures of prediction models i.e. calibration often omitted from the publication. It may therefore not be surprising that an overwhelming majority of developed prediction models are not used in practice, when there is a dearth of well-conducted and clearly reported (external validation) studies describing their performance on independent participant data.

Project description:Risk calculators can facilitate shared medical decision-making 1 . Demographics, comorbidities, medication use, geographic region, and other factors may increase the risk for COVID-19-related complications among patients with IBD 2,3 . Develop an individualized prognostic risk prediction tool for predicting the probability of adverse COVID-19 outcomes in patients with IBD. This study developed and validated prognostic penalized logistic regression models 4 using reports to Surveillance Epidemiology of Coronavirus Under Research Exclusion for Inflammatory Bowel Disease (SECURE-IBD) from March-October 2020. Model development was done using a training data set (85% of cases reported March 13 - September 15, 2020), and model validation was conducted using a test data set (the remaining 15% of cases plus all cases reported September 16-October 20, 2020). COVID-19 related:Hospitalization+: composite outcome of hospitalization, ICU admission, mechanical ventilation, or deathICU+: composite outcome of ICU admission, mechanical ventilation, or deathDeathWe assessed the resulting models' discrimination using the area under the curve (AUC) of the receiver-operator characteristic (ROC) curves and reported the corresponding 95% confidence intervals (CIs). We included 2709 cases from 59 countries (mean age 41.2 years [s.d. 18], 50.2% male). A total of 633 (24%) were hospitalized, 137 (5%) were admitted to the ICU or intubated, and 69 (3%) died. 2009 patients comprised the training set and 700 the test set.The models demonstrated excellent discrimination, with a test set AUC (95% CI) of 0.79 (0.75, 0.83) for Hospitalization+, 0.88 (0.82, 0.95) for ICU+, and 0.94 (0.89, 0.99) for Death. Age, comorbidities, corticosteroid use, and male gender were associated with higher risk of death, while use of biologic therapies was associated with a lower risk. Prognostic models can effectively predict who is at higher risk for COVID-19-related adverse outcomes in a population of IBD patients. A free online risk calculator ( https://covidibd.org/covid-19-risk-calculator/ ) is available for healthcare providers to facilitate discussion of risks due to COVID-19 with IBD patients. The tool numerically and visually summarizes the patient's probabilities of adverse outcomes and associated CIs. Helping physicians identify their highest-risk patients will be important in the coming months as cases rise in the US and worldwide. This tool can also serve as a model for risk stratification in other chronic diseases. Question: How well can a multivariable risk model predict the risk of hospitalization, intensive care unit (ICU) stay, or death due to COVID-19 in patients with inflammatory bowel disease (IBD)?Findings: Multivariable prediction models developed using data from an international voluntary registry of IBD patients and available for use online ( https://covidibd.org/ ) have very good discrimination for predicting hospitalization (Test set AUC 0.79) and excellent discrimination for ICU admission (Test set AUC 0.88) and death (Test set AUC 0.94). The models were developed with a training sample of 2009 cases and validated in an independent test sample of 700 cases comprised of a random sub-sample of cases and all cases entered in the registry during a one-month period after model development. Meaning: This risk prediction model may serve as an effective tool for healthcare providers to facilitate conversations about COVID-19-related risks with IBD patients.

Project description:ObjectivesClinical risk calculators (CRCs), such as NZRisk, are used daily by clinicians to guide clinical decisions and explain individual risk to patients. The utility and robustness of these tools depends on the methods used to create the underlying mathematical model, as well as the stability of that model in relation to changing clinical practice and patient populations over time. The later should be checked by temporal validation using external data. Few if any of the clinical prediction models in current clinical use have published temporal validation. Here, we use a large external dataset to temporally validate NZRisk; a perioperative risk prediction model used in the New Zealand population.MethodsA sample of 1 976 362 adult non-cardiac surgical procedures collected over 15 years from the New Zealand Ministry of Health National Minimum Dataset, was used to temporally validate NZRisk. We divided the dataset into 15 single year cohorts and compared 13 of these to our NZRisk model (2 years used for the model building were excluded). We compared the area under the curve (AUC) value, calibration slope and intercept for each single year cohort, to the same values produced by the data used to create NZRisk, by fitting a random effects meta-regression with each year cohort acting as a separate study point. In addition, we used two-sided t-tests to compare each measure across the cohorts.ResultsThe AUC values for the 30-day NZRisk model applied to our single year cohorts ranged from 0.918 to 0.940 (NZRisk AUC was 0.921). There were eight statistically different AUC values for the following years 2007-2009, 2016 and 2018-2021. The intercept values ranged from -0.004 to 0.007 and 7 years had statistically significant different intercepts during leave-one-out t-tests; 2007-2010, 2012, 2018 and 2021. The slope values ranged from 0.72 to 1.12 and 7 years had statistically significant different slopes during leave-one-out t-tests; 2010, 2011, 2017, 2018 and 2019-2021. The random effects meta-regression upheld our results related to AUC (0.54 (95% CI 0.40 to 0.99), I2 67.57 (95% CI 40.67 to 88.50), Cochran's Q<0.001) and slope (τ 0.14 (95% CI 0.01 to 0.23), I2 98.61 (95% CI 97.31 to 99.50), Cochran's Q<0.001) between year difference.ConclusionThe NZRisk model shows differences in AUC and slope but not intercept values over time. The biggest differences were in the calibration slope. The models maintained excellent discrimination over time as shown by the AUC values. These findings suggest we update our model in the next 5 years. To our knowledge, this is the first temporal validation of a CRC in current use.

Project description:ObjectivesDevelop an individualised prognostic risk prediction tool for predicting the probability of adverse COVID-19 outcomes in patients with inflammatory bowel disease (IBD).Design and settingThis study developed and validated prognostic penalised logistic regression models using reports to the international Surveillance Epidemiology of Coronavirus Under Research Exclusion for Inflammatory Bowel Disease voluntary registry from March to October 2020. Model development was done using a training data set (85% of cases reported 13 March-15 September 2020), and model validation was conducted using a test data set (the remaining 15% of cases plus all cases reported 16 September-20 October 2020).ParticipantsWe included 2709 cases from 59 countries (mean age 41.2 years (SD 18), 50.2% male). All submitted cases after removing duplicates were included.Primary and secondary outcome measuresCOVID-19 related: (1) Hospitalisation+: composite outcome of hospitalisation, ICU admission, mechanical ventilation or death; (2) Intensive Care Unit+ (ICU+): composite outcome of ICU admission, mechanical ventilation or death; (3) Death. We assessed the resulting models' discrimination using the area under the curve of the receiver operator characteristic curves and reported the corresponding 95% CIs.ResultsOf the submitted cases, a total of 633 (24%) were hospitalised, 137 (5%) were admitted to the ICU or intubated and 69 (3%) died. 2009 patients comprised the training set and 700 the test set. The models demonstrated excellent discrimination, with a test set area under the curve (95% CI) of 0.79 (0.75 to 0.83) for Hospitalisation+, 0.88 (0.82 to 0.95) for ICU+ and 0.94 (0.89 to 0.99) for Death. Age, comorbidities, corticosteroid use and male gender were associated with a higher risk of death, while the use of biological therapies was associated with a lower risk.ConclusionsPrognostic models can effectively predict who is at higher risk for COVID-19-related adverse outcomes in a population of patients with IBD. A free online risk calculator (https://covidibd.org/covid-19-risk-calculator/) is available for healthcare providers to facilitate discussion of risks due to COVID-19 with patients with IBD.

Project description:BackgroundPatients and caregivers express a desire for accurate prognostic information about time to institutionalization and mortality. Previous studies predicting institutionalization and mortality focused on the dementia stage. However, Alzheimer's disease (AD) is characterized by a long pre-dementia stage. Therefore, we developed prediction models to predict institutionalization and mortality along the AD continuum of cognitively normal to dementia.MethodsThis study included SCD/MCI patients (subjective cognitive decline (SCD) or mild cognitive impairment (MCI)) and patients with AD dementia from the Amsterdam Dementia Cohort. We developed internally and externally validated prediction models with biomarkers and without biomarkers, stratified by dementia status. Determinants were selected using backward selection (p<0.10). All models included age and sex. Discriminative performance of the models was assessed with Harrell's C statistics.ResultsWe included n=1418 SCD/MCI patients (n=123 died, n=74 were institutionalized) and n=1179 patients with AD dementia (n=413 died, n=453 were institutionalized). For both SCD/MCI and dementia stages, the models for institutionalization and mortality included after backward selection clinical characteristics, imaging, and cerebrospinal fluid (CSF) biomarkers. In SCD/MCI, the Harrell's C-statistics of the models were 0.81 (model without biomarkers: 0.76) for institutionalization and 0.79 (model without biomarker: 0.76) for mortality. In AD-dementia, the Harrell's C-statistics of the models were 0.68 (model without biomarkers: 0.67) for institutionalization and 0.65 (model without biomarker: 0.65) for mortality. Models based on data from amyloid-positive patients only had similar discrimination.ConclusionsWe constructed prediction models to predict institutionalization and mortality with good accuracy for SCD/MCI patients and moderate accuracy for patients with AD dementia. The developed prediction models can be used to provide patients and their caregivers with prognostic information on time to institutionalization and mortality along the cognitive continuum of AD.

Project description:Background and Objectives: Acute kidney injury (AKI) that results from ischemia is a common clinical syndrome and correlates with high morbidity and mortality among hospitalized patients. However, a clinical tool to predict mortality risk of ischemic AKI is not available. In this study, we aimed to develop and validate models to predict the 30-day and 1-year mortality risk of hospitalized patients with ischemic AKI. Methods A total of 1,836 admissions with ischemic AKI were recruited from 277,898 inpatients admitted to three affiliated tertiary general hospitals of Central South University in China between January 2015 and December 2015. Patients in the final analysis were followed up for 1 year. Study patients were randomly divided in a 7:3 ratio to form the training cohort and validation cohort. Multivariable regression analyses were used for developing mortality prediction models. Results Hepatorenal syndrome, shock, central nervous system failure, Charlson comorbidity index (≥2 points), mechanical ventilation, renal function at discharge were independent risk factors for 30-day mortality after ischemic AKI, while malignancy, sepsis, heart failure, liver failure, Charlson comorbidity index (≥2 points), mechanical ventilation, and renal function at discharge were predictors for 1-year mortality. The area under the receiver operating characteristic curves (AUROCs) of 30-day prediction model were 0.878 (95% confidence interval (CI): 0.849-0.908) in the training cohort and 0.867 (95% CI: 0.820–0.913) in the validation cohort. The AUROCs of the 1-year mortality prediction in the training and validation cohort were 0.803 (95% CI: 0.772–0.834) and 0.788 (95% CI: 0.741–0.835), respectively. Conclusion Our easily applied prediction models can effectively identify individuals at high mortality risk within 30 days or 1 year in hospitalized patients with ischemic AKI. It can guide the optimal clinical management to minimize mortality after an episode of ischemic AKI.