Development and validation of risk prediction models for COVID-19 positivity in a hospital setting.
Ontology highlight
ABSTRACT: Objectives
To develop: (1) two validated risk prediction models for coronavirus disease-2019 (COVID-19) positivity using readily available parameters in a general hospital setting; (2) nomograms and probabilities to allow clinical utilisation.Methods
Patients with and without COVID-19 were included from 4 Hong Kong hospitals. The database was randomly split into 2:1: for model development database (n = 895) and validation database (n = 435). Multivariable logistic regression was utilised for model creation and validated with the Hosmer-Lemeshow (H-L) test and calibration plot. Nomograms and probabilities set at 0.1, 0.2, 0.4 and 0.6 were calculated to determine sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV).Results
A total of 1330 patients (mean age 58.2 ± 24.5 years; 50.7% males; 296 COVID-19 positive) were recruited. The first prediction model developed had age, total white blood cell count, chest x-ray appearances and contact history as significant predictors (AUC = 0.911 [CI = 0.880-0.941]). The second model developed has the same variables except contact history (AUC = 0.880 [CI = 0.844-0.916]). Both were externally validated on the H-L test (p = 0.781 and 0.155, respectively) and calibration plot. Models were converted to nomograms. Lower probabilities give higher sensitivity and NPV; higher probabilities give higher specificity and PPV.Conclusion
Two simple-to-use validated nomograms were developed with excellent AUCs based on readily available parameters and can be considered for clinical utilisation.
SUBMITTER: Ng MY
PROVIDER: S-EPMC7491462 | biostudies-literature | 2020 Dec
REPOSITORIES: biostudies-literature
Publications
Development and validation of risk prediction models for COVID-19 positivity in a hospital setting.
Ng Ming-Yen MY Wan Eric Yuk Fai EYF Wong Ho Yuen Frank HYF Leung Siu Ting ST Lee Jonan Chun Yin JCY Chin Thomas Wing-Yan TW Lo Christine Shing Yen CSY Lui Macy Mei-Sze MM Chan Edward Hung Tat EHT Fong Ambrose Ho-Tung AH Fung Sau Yung SY Ching On Hang OH Chiu Keith Wan-Hang KW Chung Tom Wai Hin TWH Vardhanbhuti Varut V Lam Hiu Yin Sonia HYS To Kelvin Kai Wang KKW Chiu Jeffrey Long Fung JLF Lam Tina Poy Wing TPW Khong Pek Lan PL Liu Raymond Wai To RWT Chan Johnny Wai Man JWM Wu Alan Ka Lun AKL Lung Kwok-Cheung KC Hung Ivan Fan Ngai IFN Lau Chak Sing CS Kuo Michael D MD Ip Mary Sau-Man MS
International journal of infectious diseases : IJID : official publication of the International Society for Infectious Diseases 20200915
<h4>Objectives</h4>To develop: (1) two validated risk prediction models for coronavirus disease-2019 (COVID-19) positivity using readily available parameters in a general hospital setting; (2) nomograms and probabilities to allow clinical utilisation.<h4>Methods</h4>Patients with and without COVID-19 were included from 4 Hong Kong hospitals. The database was randomly split into 2:1: for model development database (n = 895) and validation database (n = 435). Multivariable logistic regression was ...[more]
Similar Datasets
Project description:BackgroundDuring the pandemic, remote consultations have become the norm for assessing patients with signs and symptoms of COVID-19 to decrease the risk of transmission. This has intensified the clinical uncertainty already experienced by primary care clinicians when assessing patients with suspected COVID-19 and has prompted the use of risk prediction scores, such as the National Early Warning Score (NEWS2), to assess severity and guide treatment. However, the risk prediction tools available have not been validated in a community setting and are not designed to capture the idiosyncrasies of COVID-19 infection.ObjectiveThe objective of this study is to produce a multivariate risk prediction tool, RECAP-V1 (Remote COVID-19 Assessment in Primary Care), to support primary care clinicians in the identification of those patients with COVID-19 that are at higher risk of deterioration and facilitate the early escalation of their treatment with the aim of improving patient outcomes.MethodsThe study follows a prospective cohort observational design, whereby patients presenting in primary care with signs and symptoms suggestive of COVID-19 will be followed and their data linked to hospital outcomes (hospital admission and death). Data collection will be carried out by primary care clinicians in four arms: North West London Clinical Commissioning Groups (NWL CCGs), Oxford-Royal College of General Practitioners (RCGP) Research and Surveillance Centre (RSC), Covid Clinical Assessment Service (CCAS), and South East London CCGs (Doctaly platform). The study involves the use of an electronic template that incorporates a list of items (known as RECAP-V0) thought to be associated with disease outcome according to previous qualitative work. Data collected will be linked to patient outcomes in highly secure environments. We will then use multivariate logistic regression analyses for model development and validation.ResultsRecruitment of participants started in October 2020. Initially, only the NWL CCGs and RCGP RSC arms were active. As of March 24, 2021, we have recruited a combined sample of 3827 participants in these two arms. CCAS and Doctaly joined the study in February 2021, with CCAS starting the recruitment process on March 15, 2021. The first part of the analysis (RECAP-V1 model development) is planned to start in April 2021 using the first half of the NWL CCGs and RCGP RSC combined data set. Posteriorly, the model will be validated with the rest of the NWL CCGs and RCGP RSC data as well as the CCAS and Doctaly data sets. The study was approved by the Research Ethics Committee on May 27, 2020 (Integrated Research Application System number: 283024, Research Ethics Committee reference number: 20/NW/0266) and badged as National Institute of Health Research Urgent Public Health Study on October 14, 2020.ConclusionsWe believe the validated RECAP-V1 early warning score will be a valuable tool for the assessment of severity in patients with suspected COVID-19 in the community, either in face-to-face or remote consultations, and will facilitate the timely escalation of treatment with the potential to improve patient outcomes.Trial registrationISRCTN registry ISRCTN13953727; https://www.isrctn.com/ISRCTN13953727.International registered report identifier (irrid)DERR1-10.2196/29072.
| S-EPMC8153031 | biostudies-literature
Project description:ObjectiveOur objective is to compare the predictive accuracy of four recently established outcome models of patients hospitalized with coronavirus disease 2019 (COVID-19) published between January 1st and May 1st 2020.MethodsWe used data obtained from the Veterans Affairs Corporate Data Warehouse (CDW) between January 1st, 2020, and May 1st 2020 as an external validation cohort. The outcome measure was hospital mortality. Areas under the ROC (AUC) curves were used to evaluate discrimination of the four predictive models. The Hosmer-Lemeshow (HL) goodness-of-fit test and calibration curves assessed applicability of the models to individual cases.ResultsDuring the study period, 1634 unique patients were identified. The mean age of the study cohort was 68.8±13.4 years. Hypertension, hyperlipidemia, and heart disease were the most common comorbidities. The crude hospital mortality was 29% (95% confidence interval [CI] 0.27-0.31). Evaluation of the predictive models showed an AUC range from 0.63 (95% CI 0.60-0.66) to 0.72 (95% CI 0.69-0.74) indicating fair to poor discrimination across all models. There were no significant differences among the AUC values of the four prognostic systems. All models calibrated poorly by either overestimated or underestimated hospital mortality.ConclusionsAll the four prognostic models examined in this study portend high-risk bias. The performance of these scores needs to be interpreted with caution in hospitalized patients with COVID-19.
| S-EPMC7769558 | biostudies-literature
Project description:BackgroundCoronavirus disease 2019 (COVID-19) carries high morbidity and mortality globally. Identification of patients at risk for clinical deterioration upon presentation would aid in triaging, prognostication, and allocation of resources and experimental treatments.Research questionCan we develop and validate a web-based risk prediction model for identification of patients who may develop severe COVID-19, defined as intensive care unit (ICU) admission, mechanical ventilation, and/or death?MethodsThis retrospective cohort study reviewed 415 patients admitted to a large urban academic medical center and community hospitals. Covariates included demographic, clinical, and laboratory data. The independent association of predictors with severe COVID-19 was determined using multivariable logistic regression. A derivation cohort (n=311, 75%) was used to develop the prediction models. The models were tested by a validation cohort (n=104, 25%).ResultsThe median age was 66 years (Interquartile range [IQR] 54-77) and the majority were male (55%) and non-White (65.8%). The 14-day severe COVID-19 rate was 39.3%; 31.7% required ICU, 24.6% mechanical ventilation, and 21.2% died. Machine learning algorithms and clinical judgment were used to improve model performance and clinical utility, resulting in the selection of eight predictors: age, sex, dyspnea, diabetes mellitus, troponin, C-reactive protein, D-dimer, and aspartate aminotransferase. The discriminative ability was excellent for both the severe COVID-19 (training area under the curve [AUC]=0.82, validation AUC=0.82) and mortality (training AUC= 0.85, validation AUC=0.81) models. These models were incorporated into a mobile-friendly website.ConclusionsThis web-based risk prediction model can be used at the bedside for prediction of severe COVID-19 using data mostly available at the time of presentation.
| S-EPMC8141270 | biostudies-literature
Project description:Risk calculators can facilitate shared medical decision-making 1 . Demographics, comorbidities, medication use, geographic region, and other factors may increase the risk for COVID-19-related complications among patients with IBD 2,3 . Develop an individualized prognostic risk prediction tool for predicting the probability of adverse COVID-19 outcomes in patients with IBD. This study developed and validated prognostic penalized logistic regression models 4 using reports to Surveillance Epidemiology of Coronavirus Under Research Exclusion for Inflammatory Bowel Disease (SECURE-IBD) from March-October 2020. Model development was done using a training data set (85% of cases reported March 13 - September 15, 2020), and model validation was conducted using a test data set (the remaining 15% of cases plus all cases reported September 16-October 20, 2020). COVID-19 related:Hospitalization+: composite outcome of hospitalization, ICU admission, mechanical ventilation, or deathICU+: composite outcome of ICU admission, mechanical ventilation, or deathDeathWe assessed the resulting models' discrimination using the area under the curve (AUC) of the receiver-operator characteristic (ROC) curves and reported the corresponding 95% confidence intervals (CIs). We included 2709 cases from 59 countries (mean age 41.2 years [s.d. 18], 50.2% male). A total of 633 (24%) were hospitalized, 137 (5%) were admitted to the ICU or intubated, and 69 (3%) died. 2009 patients comprised the training set and 700 the test set.The models demonstrated excellent discrimination, with a test set AUC (95% CI) of 0.79 (0.75, 0.83) for Hospitalization+, 0.88 (0.82, 0.95) for ICU+, and 0.94 (0.89, 0.99) for Death. Age, comorbidities, corticosteroid use, and male gender were associated with higher risk of death, while use of biologic therapies was associated with a lower risk. Prognostic models can effectively predict who is at higher risk for COVID-19-related adverse outcomes in a population of IBD patients. A free online risk calculator ( https://covidibd.org/covid-19-risk-calculator/ ) is available for healthcare providers to facilitate discussion of risks due to COVID-19 with IBD patients. The tool numerically and visually summarizes the patient's probabilities of adverse outcomes and associated CIs. Helping physicians identify their highest-risk patients will be important in the coming months as cases rise in the US and worldwide. This tool can also serve as a model for risk stratification in other chronic diseases. Question: How well can a multivariable risk model predict the risk of hospitalization, intensive care unit (ICU) stay, or death due to COVID-19 in patients with inflammatory bowel disease (IBD)?Findings: Multivariable prediction models developed using data from an international voluntary registry of IBD patients and available for use online ( https://covidibd.org/ ) have very good discrimination for predicting hospitalization (Test set AUC 0.79) and excellent discrimination for ICU admission (Test set AUC 0.88) and death (Test set AUC 0.94). The models were developed with a training sample of 2009 cases and validated in an independent test sample of 700 cases comprised of a random sub-sample of cases and all cases entered in the registry during a one-month period after model development. Meaning: This risk prediction model may serve as an effective tool for healthcare providers to facilitate conversations about COVID-19-related risks with IBD patients.
| S-EPMC7836127 | biostudies-literature
Project description:ObjectiveTo develop predictive models of coronavirus disease 2019 (COVID-19) outcomes, elucidate the influence of socioeconomic factors, and assess algorithmic racial fairness using a racially diverse patient population with high social needs.Materials and methodsData included 7,102 patients with positive (RT-PCR) severe acute respiratory syndrome coronavirus 2 test at a safety-net system in Massachusetts. Linear and nonlinear classification methods were applied. A score based on a recurrent neural network and a transformer architecture was developed to capture the dynamic evolution of vital signs. Combined with patient characteristics, clinical variables, and hospital occupancy measures, this dynamic vital score was used to train predictive models.ResultsHospitalizations can be predicted with an area under the receiver-operating characteristic curve (AUC) of 92% using symptoms, hospital occupancy, and patient characteristics, including social determinants of health. Parsimonious models to predict intensive care, mechanical ventilation, and mortality that used the most recent labs and vitals exhibited AUCs of 92.7%, 91.2%, and 94%, respectively. Early predictive models, using labs and vital signs closer to admission had AUCs of 81.1%, 84.9%, and 92%, respectively.DiscussionThe most accurate models exhibit racial bias, being more likely to falsely predict that Black patients will be hospitalized. Models that are only based on the dynamic vital score exhibited accuracies close to the best parsimonious models, although the latter also used laboratories.ConclusionsThis large study demonstrates that COVID-19 severity may accurately be predicted using a score that accounts for the dynamic evolution of vital signs. Further, race, social determinants of health, and hospital occupancy play an important role.
| S-EPMC9129120 | biostudies-literature
Project description:ObjectivesDevelop an individualised prognostic risk prediction tool for predicting the probability of adverse COVID-19 outcomes in patients with inflammatory bowel disease (IBD).Design and settingThis study developed and validated prognostic penalised logistic regression models using reports to the international Surveillance Epidemiology of Coronavirus Under Research Exclusion for Inflammatory Bowel Disease voluntary registry from March to October 2020. Model development was done using a training data set (85% of cases reported 13 March-15 September 2020), and model validation was conducted using a test data set (the remaining 15% of cases plus all cases reported 16 September-20 October 2020).ParticipantsWe included 2709 cases from 59 countries (mean age 41.2 years (SD 18), 50.2% male). All submitted cases after removing duplicates were included.Primary and secondary outcome measuresCOVID-19 related: (1) Hospitalisation+: composite outcome of hospitalisation, ICU admission, mechanical ventilation or death; (2) Intensive Care Unit+ (ICU+): composite outcome of ICU admission, mechanical ventilation or death; (3) Death. We assessed the resulting models' discrimination using the area under the curve of the receiver operator characteristic curves and reported the corresponding 95% CIs.ResultsOf the submitted cases, a total of 633 (24%) were hospitalised, 137 (5%) were admitted to the ICU or intubated and 69 (3%) died. 2009 patients comprised the training set and 700 the test set. The models demonstrated excellent discrimination, with a test set area under the curve (95% CI) of 0.79 (0.75 to 0.83) for Hospitalisation+, 0.88 (0.82 to 0.95) for ICU+ and 0.94 (0.89 to 0.99) for Death. Age, comorbidities, corticosteroid use and male gender were associated with a higher risk of death, while the use of biological therapies was associated with a lower risk.ConclusionsPrognostic models can effectively predict who is at higher risk for COVID-19-related adverse outcomes in a population of patients with IBD. A free online risk calculator (https://covidibd.org/covid-19-risk-calculator/) is available for healthcare providers to facilitate discussion of risks due to COVID-19 with patients with IBD.
| S-EPMC8593277 | biostudies-literature
Project description:ObjectiveTo provide focused evaluation of predictive modeling of electronic medical record (EMR) data to predict 30 day hospital readmission.DesignSystematic review.Data sourceOvid Medline, Ovid Embase, CINAHL, Web of Science, and Scopus from January 2015 to January 2019.Eligibility criteria for selecting studiesAll studies of predictive models for 28 day or 30 day hospital readmission that used EMR data.Outcome measuresCharacteristics of included studies, methods of prediction, predictive features, and performance of predictive models.ResultsOf 4442 citations reviewed, 41 studies met the inclusion criteria. Seventeen models predicted risk of readmission for all patients and 24 developed predictions for patient specific populations, with 13 of those being developed for patients with heart conditions. Except for two studies from the UK and Israel, all were from the US. The total sample size for each model ranged between 349 and 1 195 640. Twenty five models used a split sample validation technique. Seventeen of 41 studies reported C statistics of 0.75 or greater. Fifteen models used calibration techniques to further refine the model. Using EMR data enabled final predictive models to use a wide variety of clinical measures such as laboratory results and vital signs; however, use of socioeconomic features or functional status was rare. Using natural language processing, three models were able to extract relevant psychosocial features, which substantially improved their predictions. Twenty six studies used logistic or Cox regression models, and the rest used machine learning methods. No statistically significant difference (difference 0.03, 95% confidence interval -0.0 to 0.07) was found between average C statistics of models developed using regression methods (0.71, 0.68 to 0.73) and machine learning (0.74, 0.71 to 0.77).ConclusionsOn average, prediction models using EMR data have better predictive performance than those using administrative data. However, this improvement remains modest. Most of the studies examined lacked inclusion of socioeconomic features, failed to calibrate the models, neglected to conduct rigorous diagnostic testing, and did not discuss clinical impact.
| S-EPMC7249246 | biostudies-literature
Project description:BackgroundMost existing prognostic models of COVID-19 require imaging manifestations and laboratory results as predictors, which are only available in the post-hospitalization period. Therefore, we aimed to develop and validate a prognostic model to assess the in-hospital death risk in COVID-19 patients using routinely available predictors at hospital admission.MethodsWe conducted a retrospective cohort study of patients with COVID-19 using the Healthcare Cost and Utilization Project State Inpatient Database in 2020. Patients hospitalized in Eastern United States (Florida, Michigan, Kentucky, and Maryland) were included in the training set, and those hospitalized in Western United States (Nevada) were included in the validation set. Discrimination, calibration, and clinical utility were evaluated to assess the model's performance.ResultsA total of 17 954 in-hospital deaths occurred in the training set (n = 168 137), and 1,352 in-hospital deaths occurred in the validation set (n = 12 577). The final prediction model included 15 variables readily available at hospital admission, including age, sex, and 13 comorbidities. This prediction model showed moderate discrimination with an area under the curve (AUC) of 0.726 (95% confidence interval [CI]: 0.722-0.729) and good calibration (Brier score = 0.090, slope = 1, intercept = 0) in the training set; a similar predictive ability was observed in the validation set.ConclusionAn easy-to-use prognostic model based on predictors readily available at hospital admission was developed and validated for the early identification of COVID-19 patients with a high risk of in-hospital death. This model can be a clinical decision-support tool to triage patients and optimize resource allocation.
| S-EPMC10254410 | biostudies-literature
Project description:ObjectiveCoronavirus disease 2019 (COVID-19) patients are at risk for resource-intensive outcomes including mechanical ventilation (MV), renal replacement therapy (RRT), and readmission. Accurate outcome prognostication could facilitate hospital resource allocation. We develop and validate predictive models for each outcome using retrospective electronic health record data for COVID-19 patients treated between March 2 and May 6, 2020.Materials and methodsFor each outcome, we trained 3 classes of prediction models using clinical data for a cohort of SARS-CoV-2 (severe acute respiratory syndrome coronavirus 2)-positive patients (n = 2256). Cross-validation was used to select the best-performing models per the areas under the receiver-operating characteristic and precision-recall curves. Models were validated using a held-out cohort (n = 855). We measured each model's calibration and evaluated feature importances to interpret model output.ResultsThe predictive performance for our selected models on the held-out cohort was as follows: area under the receiver-operating characteristic curve-MV 0.743 (95% CI, 0.682-0.812), RRT 0.847 (95% CI, 0.772-0.936), readmission 0.871 (95% CI, 0.830-0.917); area under the precision-recall curve-MV 0.137 (95% CI, 0.047-0.175), RRT 0.325 (95% CI, 0.117-0.497), readmission 0.504 (95% CI, 0.388-0.604). Predictions were well calibrated, and the most important features within each model were consistent with clinical intuition.DiscussionOur models produce performant, well-calibrated, and interpretable predictions for COVID-19 patients at risk for the target outcomes. They demonstrate the potential to accurately estimate outcome prognosis in resource-constrained care sites managing COVID-19 patients.ConclusionsWe develop and validate prognostic models targeting MV, RRT, and readmission for hospitalized COVID-19 patients which produce accurate, interpretable predictions. Additional external validation studies are needed to further verify the generalizability of our results.
| S-EPMC7989331 | biostudies-literature
Project description:BackgroundThe COVID-19 pandemic has a large impact worldwide and is known to particularly affect the older population. This paper outlines the protocol for external validation of prognostic models predicting mortality risk after presentation with COVID-19 in the older population. These prognostic models were originally developed in an adult population and will be validated in an older population (≥ 70 years of age) in three healthcare settings: the hospital setting, the primary care setting, and the nursing home setting.MethodsBased on a living systematic review of COVID-19 prediction models, we identified eight prognostic models predicting the risk of mortality in adults with a COVID-19 infection (five COVID-19 specific models: GAL-COVID-19 mortality, 4C Mortality Score, NEWS2 + model, Xie model, and Wang clinical model and three pre-existing prognostic scores: APACHE-II, CURB65, SOFA). These eight models will be validated in six different cohorts of the Dutch older population (three hospital cohorts, two primary care cohorts, and a nursing home cohort). All prognostic models will be validated in a hospital setting while the GAL-COVID-19 mortality model will be validated in hospital, primary care, and nursing home settings. The study will include individuals ≥ 70 years of age with a highly suspected or PCR-confirmed COVID-19 infection from March 2020 to December 2020 (and up to December 2021 in a sensitivity analysis). The predictive performance will be evaluated in terms of discrimination, calibration, and decision curves for each of the prognostic models in each cohort individually. For prognostic models with indications of miscalibration, an intercept update will be performed after which predictive performance will be re-evaluated.DiscussionInsight into the performance of existing prognostic models in one of the most vulnerable populations clarifies the extent to which tailoring of COVID-19 prognostic models is needed when models are applied to the older population. Such insight will be important for possible future waves of the COVID-19 pandemic or future pandemics.
| S-EPMC10069944 | biostudies-literature