Dataset Information

Development and validation of multivariable mortality risk-prediction models in older people undergoing an interRAI home-care assessment (RiskOP)

ABSTRACT:

Background

Currently, one-year survival of older people with complex co-morbidities is unpredictable. Identifying older adults with a reduced life expectancy will lead to more targeted care and better healthcare resource allocation.

Methods

Development and validation of one-year and three-month mortality risks in people aged ?65 years who had completed an International Resident Assessment Instrument-Home Care (interRAI-HC) assessment between July 2012 and March 2018. Data was split into development (90%) and validation data sets (10%). A multivariable logistic regression model using data from 108 interRAI questions across multiple domains was developed and validated using discrimination metrics and calibration curves. Variables each explaining at least 1% of the model were then used to develop and validate a parsimonious model. Subgroups by sex, age, ethnicity, and comorbidities were evaluated.

Findings

There were 104,436 persons (60.2% female; mean age 82.1 years) in the study cohort of whom 20,972 (20.1%) died within one year. The full multivariable model had area under the curves (AUCs) of 0.778 to 0.795 in the 5 validation datasets and was well calibrated. After variable reduction a parsimonious model consisted of 16 variables and was well calibrated and the AUC remained high: 0.773 (0.769 to 0.777). The three-month parsimonious model comprised 22 variables and was well calibrated with an AUC of 0.843 (95%CI: 0.839 to 0.848).

Interpretation

These community-based risk prediction models accurately predict mortality in older people with complex co-morbidities. They may contribute to both forecasting for policy making and clinical decision making regarding an individual's needs.

Funding

The New Zealand Health Research Council.

SUBMITTER: Pickering J

PROVIDER: S-EPMC7788437 | biostudies-literature | 2020 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundBefore considering whether to use a multivariable (diagnostic or prognostic) prediction model, it is essential that its performance be evaluated in data that were not used to develop the model (referred to as external validation). We critically appraised the methodological conduct and reporting of external validation studies of multivariable prediction models.MethodsWe conducted a systematic review of articles describing some form of external validation of one or more multivariable prediction models indexed in PubMed core clinical journals published in 2010. Study data were extracted in duplicate on design, sample size, handling of missing data, reference to the original study developing the prediction models and predictive performance measures.Results11,826 articles were identified and 78 were included for full review, which described the evaluation of 120 prediction models. in participant data that were not used to develop the model. Thirty-three articles described both the development of a prediction model and an evaluation of its performance on a separate dataset, and 45 articles described only the evaluation of an existing published prediction model on another dataset. Fifty-seven percent of the prediction models were presented and evaluated as simplified scoring systems. Sixteen percent of articles failed to report the number of outcome events in the validation datasets. Fifty-four percent of studies made no explicit mention of missing data. Sixty-seven percent did not report evaluating model calibration whilst most studies evaluated model discrimination. It was often unclear whether the reported performance measures were for the full regression model or for the simplified models.ConclusionsThe vast majority of studies describing some form of external validation of a multivariable prediction model were poorly reported with key details frequently not presented. The validation studies were characterised by poor design, inappropriate handling and acknowledgement of missing data and one of the most key performance measures of prediction models i.e. calibration often omitted from the publication. It may therefore not be surprising that an overwhelming majority of developed prediction models are not used in practice, when there is a dearth of well-conducted and clearly reported (external validation) studies describing their performance on independent participant data.

Project description:Risk calculators can facilitate shared medical decision-making 1 . Demographics, comorbidities, medication use, geographic region, and other factors may increase the risk for COVID-19-related complications among patients with IBD 2,3 . Develop an individualized prognostic risk prediction tool for predicting the probability of adverse COVID-19 outcomes in patients with IBD. This study developed and validated prognostic penalized logistic regression models 4 using reports to Surveillance Epidemiology of Coronavirus Under Research Exclusion for Inflammatory Bowel Disease (SECURE-IBD) from March-October 2020. Model development was done using a training data set (85% of cases reported March 13 - September 15, 2020), and model validation was conducted using a test data set (the remaining 15% of cases plus all cases reported September 16-October 20, 2020). COVID-19 related:Hospitalization+: composite outcome of hospitalization, ICU admission, mechanical ventilation, or deathICU+: composite outcome of ICU admission, mechanical ventilation, or deathDeathWe assessed the resulting models' discrimination using the area under the curve (AUC) of the receiver-operator characteristic (ROC) curves and reported the corresponding 95% confidence intervals (CIs). We included 2709 cases from 59 countries (mean age 41.2 years [s.d. 18], 50.2% male). A total of 633 (24%) were hospitalized, 137 (5%) were admitted to the ICU or intubated, and 69 (3%) died. 2009 patients comprised the training set and 700 the test set.The models demonstrated excellent discrimination, with a test set AUC (95% CI) of 0.79 (0.75, 0.83) for Hospitalization+, 0.88 (0.82, 0.95) for ICU+, and 0.94 (0.89, 0.99) for Death. Age, comorbidities, corticosteroid use, and male gender were associated with higher risk of death, while use of biologic therapies was associated with a lower risk. Prognostic models can effectively predict who is at higher risk for COVID-19-related adverse outcomes in a population of IBD patients. A free online risk calculator ( https://covidibd.org/covid-19-risk-calculator/ ) is available for healthcare providers to facilitate discussion of risks due to COVID-19 with IBD patients. The tool numerically and visually summarizes the patient's probabilities of adverse outcomes and associated CIs. Helping physicians identify their highest-risk patients will be important in the coming months as cases rise in the US and worldwide. This tool can also serve as a model for risk stratification in other chronic diseases. Question: How well can a multivariable risk model predict the risk of hospitalization, intensive care unit (ICU) stay, or death due to COVID-19 in patients with inflammatory bowel disease (IBD)?Findings: Multivariable prediction models developed using data from an international voluntary registry of IBD patients and available for use online ( https://covidibd.org/ ) have very good discrimination for predicting hospitalization (Test set AUC 0.79) and excellent discrimination for ICU admission (Test set AUC 0.88) and death (Test set AUC 0.94). The models were developed with a training sample of 2009 cases and validated in an independent test sample of 700 cases comprised of a random sub-sample of cases and all cases entered in the registry during a one-month period after model development. Meaning: This risk prediction model may serve as an effective tool for healthcare providers to facilitate conversations about COVID-19-related risks with IBD patients.

Project description:ObjectivesClinical risk calculators (CRCs), such as NZRisk, are used daily by clinicians to guide clinical decisions and explain individual risk to patients. The utility and robustness of these tools depends on the methods used to create the underlying mathematical model, as well as the stability of that model in relation to changing clinical practice and patient populations over time. The later should be checked by temporal validation using external data. Few if any of the clinical prediction models in current clinical use have published temporal validation. Here, we use a large external dataset to temporally validate NZRisk; a perioperative risk prediction model used in the New Zealand population.MethodsA sample of 1 976 362 adult non-cardiac surgical procedures collected over 15 years from the New Zealand Ministry of Health National Minimum Dataset, was used to temporally validate NZRisk. We divided the dataset into 15 single year cohorts and compared 13 of these to our NZRisk model (2 years used for the model building were excluded). We compared the area under the curve (AUC) value, calibration slope and intercept for each single year cohort, to the same values produced by the data used to create NZRisk, by fitting a random effects meta-regression with each year cohort acting as a separate study point. In addition, we used two-sided t-tests to compare each measure across the cohorts.ResultsThe AUC values for the 30-day NZRisk model applied to our single year cohorts ranged from 0.918 to 0.940 (NZRisk AUC was 0.921). There were eight statistically different AUC values for the following years 2007-2009, 2016 and 2018-2021. The intercept values ranged from -0.004 to 0.007 and 7 years had statistically significant different intercepts during leave-one-out t-tests; 2007-2010, 2012, 2018 and 2021. The slope values ranged from 0.72 to 1.12 and 7 years had statistically significant different slopes during leave-one-out t-tests; 2010, 2011, 2017, 2018 and 2019-2021. The random effects meta-regression upheld our results related to AUC (0.54 (95% CI 0.40 to 0.99), I2 67.57 (95% CI 40.67 to 88.50), Cochran's Q<0.001) and slope (τ 0.14 (95% CI 0.01 to 0.23), I2 98.61 (95% CI 97.31 to 99.50), Cochran's Q<0.001) between year difference.ConclusionThe NZRisk model shows differences in AUC and slope but not intercept values over time. The biggest differences were in the calibration slope. The models maintained excellent discrimination over time as shown by the AUC values. These findings suggest we update our model in the next 5 years. To our knowledge, this is the first temporal validation of a CRC in current use.

Project description:Background and aimsFew studies have meta-analyzed different prognostic models developed for older adults, especially nursing home residents. We aimed to systematically review and meta-analyze the performance of all published models that predicted all-cause mortality among older nursing home residents.MethodsWe systematically searched PubMed and EMBASE from the databases' inception to January 1, 2020 to capture studies developing and/or validating a prognostic/prediction model for all-cause mortality among nursing home residents. We then carried out both qualitative and quantitative analyses evaluating these models' risks of bias and applicability.ResultsThe systematic search yielded 23,975 articles. We identified 28 indices that predicted the risk of all-cause mortality from 14 days to 39 months among older adults in nursing homes. The most used predictors were age, sex, body weight, swallowing problem, congestive heart failure, shortness of breath, body mass index, and activities of daily living. Of the 28 indices, 8 (29%) and 3 (11%) were internally and externally validated, respectively. None of the indices was validated in more than one cohort. Of the 28 indices, 22 (79%) reported the C-statistic, while only 6 (6%) reported the 95% confidence interval for the C statistic in the development cohorts. In the validation cohorts, 11 (39%) reported the C-statistic and 8 (29%) reported the 95% confidence interval. The meta-analyzed C statistic for all indices is 0.733 (95% prediction interval: 0.669-0.797). All studies/indices had high risks of bias and high concern for applicability according to PROBAST.ConclusionWe identified 28 indices for predicting all-cause mortality among older nursing home residents. The overall quality of evidence was low due to a high degree of bias and poor reporting of model performance statistics. Before any prediction model could be recommended in routine care, future research is needed to rigorously validate existing prediction models and evaluate their applicability and develop new prediction models.

Project description:ObjectivesDevelop an individualised prognostic risk prediction tool for predicting the probability of adverse COVID-19 outcomes in patients with inflammatory bowel disease (IBD).Design and settingThis study developed and validated prognostic penalised logistic regression models using reports to the international Surveillance Epidemiology of Coronavirus Under Research Exclusion for Inflammatory Bowel Disease voluntary registry from March to October 2020. Model development was done using a training data set (85% of cases reported 13 March-15 September 2020), and model validation was conducted using a test data set (the remaining 15% of cases plus all cases reported 16 September-20 October 2020).ParticipantsWe included 2709 cases from 59 countries (mean age 41.2 years (SD 18), 50.2% male). All submitted cases after removing duplicates were included.Primary and secondary outcome measuresCOVID-19 related: (1) Hospitalisation+: composite outcome of hospitalisation, ICU admission, mechanical ventilation or death; (2) Intensive Care Unit+ (ICU+): composite outcome of ICU admission, mechanical ventilation or death; (3) Death. We assessed the resulting models' discrimination using the area under the curve of the receiver operator characteristic curves and reported the corresponding 95% CIs.ResultsOf the submitted cases, a total of 633 (24%) were hospitalised, 137 (5%) were admitted to the ICU or intubated and 69 (3%) died. 2009 patients comprised the training set and 700 the test set. The models demonstrated excellent discrimination, with a test set area under the curve (95% CI) of 0.79 (0.75 to 0.83) for Hospitalisation+, 0.88 (0.82 to 0.95) for ICU+ and 0.94 (0.89 to 0.99) for Death. Age, comorbidities, corticosteroid use and male gender were associated with a higher risk of death, while the use of biological therapies was associated with a lower risk.ConclusionsPrognostic models can effectively predict who is at higher risk for COVID-19-related adverse outcomes in a population of patients with IBD. A free online risk calculator (https://covidibd.org/covid-19-risk-calculator/) is available for healthcare providers to facilitate discussion of risks due to COVID-19 with patients with IBD.

Project description:Two neonatal mortality prediction models, the Neonatal Essential Treatment Score (NETS) which uses treatments prescribed at admission and the Score for Essential Neonatal Symptoms and Signs (SENSS) which uses basic clinical signs, were derived in high-mortality, low-resource settings to utilise data more likely to be available in these settings. In this study, we evaluate the predictive accuracy of two neonatal prediction models for all-cause in-hospital mortality. We used retrospectively collected routine clinical data recorded by duty clinicians at admission from 16 Kenyan hospitals used to externally validate and update the SENSS and NETS models that were initially developed from the data from the largest Kenyan maternity hospital to predict in-hospital mortality. Model performance was evaluated by assessing discrimination and calibration. Discrimination, the ability of the model to differentiate between those with and without the outcome, was measured using the c-statistic. Calibration, the agreement between predictions from the model and what was observed, was measured using the calibration intercept and slope (with values of 0 and 1 denoting perfect calibration). At initial external validation, the estimated mortality risks from the original SENSS and NETS models were markedly overestimated with calibration intercepts of - 0.703 (95% CI - 0.738 to - 0.669) and - 1.109 (95% CI - 1.148 to - 1.069) and too extreme with calibration slopes of 0.565 (95% CI 0.552 to 0.577) and 0.466 (95% CI 0.451 to 0.480), respectively. After model updating, the calibration of the model improved. The updated SENSS and NETS models had calibration intercepts of 0.311 (95% CI 0.282 to 0.350) and 0.032 (95% CI - 0.002 to 0.066) and calibration slopes of 1.029 (95% CI 1.006 to 1.051) and 0.799 (95% CI 0.774 to 0.823), respectively, while showing good discrimination with c-statistics of 0.834 (95% CI 0.829 to 0.839) and 0.775 (95% CI 0.768 to 0.782), respectively. The overall calibration performance of the updated SENSS and NETS models was better than any existing neonatal in-hospital mortality prediction models externally validated for settings comparable to Kenya. Few prediction models undergo rigorous external validation. We show how external validation using data from multiple locations enables model updating and improving their performance and potential value. The improved models indicate it is possible to predict in-hospital mortality using either treatments or signs and symptoms derived from routine neonatal data from low-resource hospital settings also making possible their use for case-mix adjustment when contrasting similar hospital settings.

Project description:BackgroundPatients and caregivers express a desire for accurate prognostic information about time to institutionalization and mortality. Previous studies predicting institutionalization and mortality focused on the dementia stage. However, Alzheimer's disease (AD) is characterized by a long pre-dementia stage. Therefore, we developed prediction models to predict institutionalization and mortality along the AD continuum of cognitively normal to dementia.MethodsThis study included SCD/MCI patients (subjective cognitive decline (SCD) or mild cognitive impairment (MCI)) and patients with AD dementia from the Amsterdam Dementia Cohort. We developed internally and externally validated prediction models with biomarkers and without biomarkers, stratified by dementia status. Determinants were selected using backward selection (p<0.10). All models included age and sex. Discriminative performance of the models was assessed with Harrell's C statistics.ResultsWe included n=1418 SCD/MCI patients (n=123 died, n=74 were institutionalized) and n=1179 patients with AD dementia (n=413 died, n=453 were institutionalized). For both SCD/MCI and dementia stages, the models for institutionalization and mortality included after backward selection clinical characteristics, imaging, and cerebrospinal fluid (CSF) biomarkers. In SCD/MCI, the Harrell's C-statistics of the models were 0.81 (model without biomarkers: 0.76) for institutionalization and 0.79 (model without biomarker: 0.76) for mortality. In AD-dementia, the Harrell's C-statistics of the models were 0.68 (model without biomarkers: 0.67) for institutionalization and 0.65 (model without biomarker: 0.65) for mortality. Models based on data from amyloid-positive patients only had similar discrimination.ConclusionsWe constructed prediction models to predict institutionalization and mortality with good accuracy for SCD/MCI patients and moderate accuracy for patients with AD dementia. The developed prediction models can be used to provide patients and their caregivers with prognostic information on time to institutionalization and mortality along the cognitive continuum of AD.