Use and Customization of Risk Scores for Predicting Cardiovascular Events Using Electronic Health Record Data.
Ontology highlight
ABSTRACT: Background
Clinicians who are using the Framingham Risk Score (FRS) or the American College of Cardiology/American Heart Association Pooled Cohort Equations (PCE) to estimate risk for their patients based on electronic health data (EHD) face 4 questions. (1) Do published risk scores applied to EHD yield accurate estimates of cardiovascular risk? (2) Are FRS risk estimates, which are based on data that are up to 45 years old, valid for a contemporary patient population seeking routine care? (3) Do the PCE make the FRS obsolete? (4) Does refitting the risk score using EHD improve the accuracy of risk estimates?Methods and results
Data were extracted from the EHD of 84 116 adults aged 40 to 79 years who received care at a large healthcare delivery and insurance organization between 2001 and 2011. We assessed calibration and discrimination for 4 risk scores: published versions of FRS and PCE and versions obtained by refitting models using a subset of the available EHD. The published FRS was well calibrated (calibration statistic K=9.1, miscalibration ranging from 0% to 17% across risk groups), but the PCE displayed modest evidence of miscalibration (calibration statistic K=43.7, miscalibration from 9% to 31%). Discrimination was similar in both models (C-index=0.740 for FRS, 0.747 for PCE). Refitting the published models using EHD did not substantially improve calibration or discrimination.Conclusions
We conclude that published cardiovascular risk models can be successfully applied to EHD to estimate cardiovascular risk; the FRS remains valid and is not obsolete; and model refitting does not meaningfully improve the accuracy of risk estimates.
SUBMITTER: Wolfson J
PROVIDER: S-EPMC5532984 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BackgroundPatients with peripheral artery disease (PAD) are at risk of major adverse cardiac and cerebrovascular events. There are no readily available risk scores that can accurately identify which patients are most likely to sustain an event, making it difficult to identify those who might benefit from more aggressive intervention. Thus, we aimed to develop a novel predictive model-using machine learning methods on electronic health record data-to identify which PAD patients are most likely to develop major adverse cardiac and cerebrovascular events.Methods and resultsData were derived from patients diagnosed with PAD at 2 tertiary care institutions. Predictive models were built using a common data model that allowed for utilization of both structured (coded) and unstructured (text) data. Only data from time of entry into the health system up to PAD diagnosis were used for modeling. Models were developed and tested using nested cross-validation. A total of 7686 patients were included in learning our predictive models. Utilizing almost 1000 variables, our best predictive model accurately determined which PAD patients would go on to develop major adverse cardiac and cerebrovascular events with an area under the curve of 0.81 (95% CI, 0.80-0.83).ConclusionsMachine learning algorithms applied to data in the electronic health record can learn models that accurately identify PAD patients at risk of future major adverse cardiac and cerebrovascular events, highlighting the great potential of electronic health records to provide automated risk stratification for cardiovascular diseases. Common data models that can enable cross-institution research and technology development could potentially be an important aspect of widespread adoption of newer risk-stratification models.
| S-EPMC6415677 | biostudies-literature
Project description:In a general inpatient population, we predicted patient-specific medication orders based on structured information in the electronic health record (EHR). Data on over three million medication orders from an academic medical center were used to train two machine-learning models: A deep learning sequence model and a logistic regression model. Both were compared with a baseline that ranked the most frequently ordered medications based on a patient's discharge hospital service and amount of time since admission. Models were trained to predict from 990 possible medications at the time of order entry. Fifty-five percent of medications ordered by physicians were ranked in the sequence model's top-10 predictions (logistic model: 49%) and 75% ranked in the top-25 (logistic model: 69%). Ninety-three percent of the sequence model's top-10 prediction sets contained at least one medication that physicians ordered within the next day. These findings demonstrate that medication orders can be predicted from information present in the EHR.
| S-EPMC7325318 | biostudies-literature
Project description:BACKGROUND:Readmissions after hospitalization for pneumonia are common, but the few risk-prediction models have poor to modest predictive ability. Data routinely collected in the electronic health record (EHR) may improve prediction. OBJECTIVE:To develop pneumonia-specific readmission risk-prediction models using EHR data from the first day and from the entire hospital stay ("full stay"). DESIGN:Observational cohort study using stepwise-backward selection and cross-validation. SUBJECTS:Consecutive pneumonia hospitalizations from 6 diverse hospitals in north Texas from 2009-2010. MEASURES:All-cause nonelective 30-day readmissions, ascertained from 75 regional hospitals. RESULTS:Of 1463 patients, 13.6% were readmitted. The first-day pneumonia-specific model included sociodemographic factors, prior hospitalizations, thrombocytosis, and a modified pneumonia severity index; the full-stay model included disposition status, vital sign instabilities on discharge, and an updated pneumonia severity index calculated using values from the day of discharge as additional predictors. The full-stay pneumonia-specific model outperformed the first-day model (C statistic 0.731 vs 0.695; P = 0.02; net reclassification index = 0.08). Compared to a validated multi-condition readmission model, the Centers for Medicare and Medicaid Services pneumonia model, and 2 commonly used pneumonia severity of illness scores, the full-stay pneumonia-specific model had better discrimination (C statistic range 0.604-0.681; P < 0.01 for all comparisons), predicted a broader range of risk, and better reclassified individuals by their true risk (net reclassification index range, 0.09-0.18). CONCLUSIONS:EHR data collected from the entire hospitalization can accurately predict readmission risk among patients hospitalized for pneumonia. This approach outperforms a first-day pneumonia-specific model, the Centers for Medicare and Medicaid Services pneumonia model, and 2 commonly used pneumonia severity of illness scores. Journal of Hospital Medicine 2017;12:209-216.
| S-EPMC6296251 | biostudies-literature
Project description:ImportanceSecondary prevention interventions to reduce post-stroke cognitive impairment (PSCI) can be aided by the early identification of high-risk individuals who would benefit from risk factor modification.ObjectiveTo develop and evaluate a predictive model to identify patients at increased risk of PSCI over 5 years using data easily accessible from electronic health records.DesignCohort study with patients enrolled between 2003-2016 with follow-up through 2022.SettingPrimary care practices affiliated with two academic medical centers.ParticipantsIndividuals 45 years or older, without prior stroke or prevalent cognitive impairment, with primary care visits and an incident ischemic stroke between 2003-2016 (development/internal validation cohort) or 2010-2022 (external validation cohort).ExposuresPredictors of PSCI were ascertained from the electronic health record.Main outcomeThe outcome was incident dementia/cognitive impairment within 5 years and beginning 3 months following stroke, ascertained using ICD-9/10 codes. For model variable selection, we considered potential predictors of PSCI and constructed 400 bootstrap samples with two-thirds of the model derivation sample. We ran 10-fold cross-validated Cox proportional hazards models using a least absolute shrinkage and selection operator (LASSO) penalty. Variables selected in >25% of samples were included.ResultsThe analysis included 332 incident diagnoses of PSCI in the development cohort (n=3,741), and 161 and 128 incident diagnoses in the internal (n=1,925) and external (n=2,237) validation cohorts. The c-statistic for predicting PSCI was 0.731 (95% CI: 0.694-0.768) in the internal validation cohort, and 0.724 (95% CI: 0.681-0.766) in the external validation cohort. A risk score based on the beta coefficients of predictors from the development cohort stratified patients into low (0-7 points), intermediate (8-11 points), and high (12-35 points) risk groups. The hazard ratios for incident PSCI were significantly different by risk categories in internal (High, HR: 6.2, 95% CI 4.1-9.3; Intermediate, HR 2.7, 95% CI: 1.8-4.1) and external (High, HR: 6.1, 95% CI: 3.9-9.6; Intermediate, HR 2.8, 95% CI: 1.9-4.3) validation cohorts.Conclusions and relevanceFive-year risk of PSCI can be accurately predicted using routinely collected data. Model output can be used to risk stratify and identify individuals at increased risk for PSCI for preventive efforts.
| S-EPMC10863024 | biostudies-literature
Project description:Predictive analytics with electronic health record (EHR) data holds promise for improving outcomes of psychiatric care. This study evaluated models for predicting outcomes of psychotherapy for depression in a clinical practice setting. EHR data from two large integrated health systems (Kaiser Permanente Colorado and Washington) included 5,554 new psychotherapy episodes with a baseline Patient Health Questionnaire (PHQ-9) score ≥ 10 and a follow-up PHQ-9 14-180 days after treatment initiation. Baseline predictors included demographics and diagnostic, medication, and encounter history. Prediction models for two outcomes-follow-up PHQ-9 score and treatment response (≥ 50% PHQ-9 reduction)-were trained in a random sample of 70% of episodes and validated in the remaining 30%. Two methods were used for modeling: generalized linear regression models with variable selection and random forests. Sensitivity analyses considered alternate predictor, outcome, and model specifications. Predictions of follow-up PHQ-9 scores poorly estimated observed outcomes (mean squared error = 31 for linear regression, 40 for random forest). Predictions of treatment response had low discrimination (AUC = 0.57 for logistic regression, 0.61 for random forest), low classification accuracy, and poor calibration. Sensitivity analyses showed similar results. We note that prediction model performance may vary for settings with different care or EHR documentation practices. In conclusion, prediction models did not accurately predict depression treatment outcomes despite using rich EHR data and advanced analytic techniques. Health systems should proceed cautiously when considering prediction models for psychiatric outcomes using baseline intake information. Transparent research should be conducted to evaluate performance of any model intended for clinical use.
| S-EPMC8448296 | biostudies-literature
Project description:Prediction of major adverse kidney events in critically ill patients may help target therapy, allow risk adjustment, and facilitate the conduct of clinical trials. In a cohort comprised of all critically ill adults admitted to five intensive care units at a single tertiary care center over one year, we developed a logistic regression model for the outcome of Major Adverse Kidney Events within 30 days (MAKE30), the composite of persistent renal dysfunction, new renal replacement therapy (RRT), and in-hospital mortality. Proposed risk factors for the MAKE30 outcome were selected a priori and included age, race, gender, University Health System Consortium (UHC) expected mortality, baseline creatinine, volume of isotonic crystalloid fluid received in the prior 24 h, admission service, intensive care unit (ICU), source of admission, mechanical ventilation or receipt of vasopressors within 24 h of ICU admission, renal replacement therapy prior to ICU admission, acute kidney injury, chronic kidney disease as defined by baseline creatinine value, and renal failure as defined by the Elixhauser index. Among 10,983 patients in the study population, 1489 patients (13.6%) met the MAKE30 endpoint. The strongest independent predictors of MAKE30 were UHC expected mortality (OR 2.32 [95%CI 2.06-2.61]) and presence of acute kidney injury at ICU admission (OR 4.98 [95%CI 4.12-6.03]). The model had strong predictive properties including excellent discrimination with a bootstrap-corrected area-under-the-curve (AUC) of 0.903, and high precision of calibration with a mean absolute error prediction of 1.7%. The MAKE30 composite outcome can be reliably predicted from factors present within 24 h of ICU admission using data derived from the electronic health record.
| S-EPMC5821255 | biostudies-literature
Project description:ImportanceData from electronic health records (EHRs) are increasingly used for risk prediction. However, EHRs do not reliably collect sociodemographic and neighborhood information, which has been shown to be associated with health. The added contribution of neighborhood socioeconomic status (nSES) in predicting health events is unknown and may help inform population-level risk reduction strategies.ObjectiveTo quantify the association of nSES with adverse outcomes and the value of nSES in predicting the risk of adverse outcomes in EHR-based risk models.Design, setting, and participantsCohort study in which data from 90 097 patients 18 years or older in the Duke University Health System and Lincoln Community Health Center EHR from January 1, 2009, to December 31, 2015, with at least 1 health care encounter and residence in Durham County, North Carolina, in the year prior to the index date were linked with census tract data to quantify the association between nSES and the risk of adverse outcomes. Machine learning methods were used to develop risk models and determine how adding nSES to EHR data affects risk prediction. Neighborhood socioeconomic status was defined using the Agency for Healthcare Research and Quality SES index, a weighted measure of multiple indicators of neighborhood deprivation.Main outcomes and measuresOutcomes included use of health care services (emergency department and inpatient and outpatient encounters) and hospitalizations due to accidents, asthma, influenza, myocardial infarction, and stroke.ResultsAmong the 90 097 patients in the training set of the study (57 507 women and 32 590 men; mean [SD] age, 47.2 [17.7] years) and the 122 812 patients in the testing set of the study (75 517 women and 47 295 men; mean [SD] age, 46.2 [17.9] years), those living in neighborhoods with lower nSES had a shorter time to use of emergency department services and inpatient encounters, as well as a shorter time to hospitalizations due to accidents, asthma, influenza, myocardial infarction, and stroke. The predictive value of nSES varied by outcome of interest (C statistic ranged from 0.50 to 0.63). When added to EHR variables, nSES did not improve predictive performance for any health outcome.Conclusions and relevanceSocial determinants of health, including nSES, are associated with the health of a patient. However, the results of this study suggest that information on nSES may not contribute much more to risk prediction above and beyond what is already provided by EHR data. Although this result does not mean that integrating social determinants of health into the EHR has no benefit, researchers may be able to use EHR data alone for population risk assessment.
| S-EPMC6324505 | biostudies-literature
Project description:Use of the electronic health record (EHR) is expected to increase rapidly in the near future, yet little research exists on whether analyzing internal EHR data using flexible, adaptive statistical methods could improve clinical risk prediction. Extensive implementation of EHR in the Veterans Health Administration provides an opportunity for exploration.To compare the performance of various approaches for predicting risk of cerebrovascular and cardiovascular (CCV) death, using traditional risk predictors versus more comprehensive EHR data.Retrospective cohort study. We identified all Veterans Health Administration patients without recent CCV events treated at 12 facilities from 2003 to 2007, and predicted risk using the Framingham risk score, logistic regression, generalized additive modeling, and gradient tree boosting.The outcome was CCV-related death within 5 years. We assessed each method's predictive performance with the area under the receiver operating characteristic curve (AUC), the Hosmer-Lemeshow goodness-of-fit test, plots of estimated risk, and reclassification tables, using cross-validation to penalize overfitting.Regression methods outperformed the Framingham risk score, even with the same predictors (AUC increased from 71% to 73% and calibration also improved). Even better performance was attained in models using additional EHR-derived predictor variables (AUC increased to 78% and net reclassification improvement was as large as 0.29). Nonparametric regression further improved calibration and discrimination compared with logistic regression.Despite the EHR lacking some risk factors and its imperfect data quality, health care systems may be able to substantially improve risk prediction for their patients by using internally developed EHR-derived models and flexible statistical methodology.
| S-EPMC4081533 | biostudies-literature
Project description:ObjectiveOutpatient clinics lack guidance for tackling modern efficiency and productivity demands. Workflow studies require large amounts of timing data that are prohibitively expensive to collect through observation or tracking devices. Electronic health records (EHRs) contain a vast amount of timing data - timestamps collected during regular use - that can be mapped to workflow steps. This study validates using EHR timestamp data to predict outpatient ophthalmology clinic workflow timings at Oregon Health and Science University and demonstrates their usefulness in 3 different studies.Materials and methodsFour outpatient ophthalmology clinics were observed to determine their workflows and to time each workflow step. EHR timestamps were mapped to the workflow steps and validated against the observed timings.ResultsThe EHR timestamp analysis produced times that were within 3 min of the observed times for >80% of the appointments. EHR use patterns affected the accuracy of using EHR timestamps to predict workflow times.DiscussionEHR timestamps provided a reasonable approximation of workflow and can be used for workflow studies. They can be used to create simulation models, analyze EHR use, and quantify the impact of trainees on workflow.ConclusionThe secondary use of EHR timestamp data is a valuable resource for clinical workflow studies. Sample timestamp data files and algorithms for processing them are provided and can be used as a template for more studies in other clinical specialties and settings.
| S-EPMC6080808 | biostudies-literature
Project description:ObjectiveTo assess whether electronic health record (EHR) data text mining can be used to improve register-based heart failure (HF) subtyping. EHR data of 43,405 individuals from two Finnish hospital biobanks were mined for unstructured text mentions of ejection fraction (EF) and validated against clinical assessment in two sets of 100 randomly selected individuals. Structured laboratory data was then incorporated for a categorization by HF subtype (HF with mildly reduced EF, HFmrEF; HF with preserved EF, HFpEF; HF with reduced EF, HFrEF; and no HF).ResultsIn 86% of the cases, the algorithm-identified EF belonged to the correct HF subtype range. Sensitivity, specificity, PPV and NPV of the algorithm were 94-100% for HFrEF, 85-100% for HFmrEF, and 96%, 67%, 53% and 98% for HFpEF. Survival analyses using the traditional diagnosis of HF were in concordance with the algorithm-based ones. Compared to healthy individuals, mortality increased from HFmrEF (hazard ratio [HR], 1.91; 95% confidence interval [CI], 1.24-2.95) to HFpEF (2.28; 1.80-2.88) to HFrEF group (2.63; 1.97-3.50) over a follow-up of 1.5 years. We conclude that quantitative EF data can be efficiently extracted from EHRs and used with laboratory data to subtype HF with reasonable accuracy, especially for HFrEF.
| S-EPMC10496250 | biostudies-literature