Dataset Information

A strategy for validation of variables derived from large-scale electronic health record data.

ABSTRACT:

Purpose

Standardized approaches for rigorous validation of phenotyping from large-scale electronic health record (EHR) data have not been widely reported. We proposed a methodologically rigorous and efficient approach to guide such validation, including strategies for sampling cases and controls, determining sample sizes, estimating algorithm performance, and terminating the validation process, hereafter referred to as the San Diego Approach to Variable Validation (SDAVV).

Methods

We propose sample size formulae which should be used prior to chart review, based on pre-specified critical lower bounds for positive predictive value (PPV) and negative predictive value (NPV). We also propose a stepwise strategy for iterative algorithm development/validation cycles, updating sample sizes for data abstraction until both PPV and NPV achieve target performance.

Results

We applied the SDAVV to a Department of Veterans Affairs study in which we created two phenotyping algorithms, one for distinguishing normal colonoscopy cases from abnormal colonoscopy controls and one for identifying aspirin exposure. Estimated PPV and NPV both reached 0.970 with a 95% confidence lower bound of 0.915, estimated sensitivity was 0.963 and specificity was 0.975 for identifying normal colonoscopy cases. The phenotyping algorithm for identifying aspirin exposure reached a PPV of 0.990 (a 95% lower bound of 0.950), an NPV of 0.980 (a 95% lower bound of 0.930), and sensitivity and specificity were 0.960 and 1.000.

Conclusions

A structured approach for prospectively developing and validating phenotyping algorithms from large-scale EHR data can be successfully implemented, and should be considered to improve the quality of "big data" research.

SUBMITTER: Liu L

PROVIDER: S-EPMC9615095 | biostudies-literature | 2021 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A strategy for validation of variables derived from large-scale electronic health record data.

Liu Lin L Bustamante Ranier R Earles Ashley A Demb Joshua J Messer Karen K Gupta Samir S

Journal of biomedical informatics 20210727

<h4>Purpose</h4>Standardized approaches for rigorous validation of phenotyping from large-scale electronic health record (EHR) data have not been widely reported. We proposed a methodologically rigorous and efficient approach to guide such validation, including strategies for sampling cases and controls, determining sample sizes, estimating algorithm performance, and terminating the validation process, hereafter referred to as the San Diego Approach to Variable Validation (SDAVV).<h4>Methods</h4 ...[more]

PMID: 34329789

Similar Datasets

Project description:Backgroundfrailty is an especially problematic expression of population ageing. International guidelines recommend routine identification of frailty to provide evidence-based treatment, but currently available tools require additional resource.Objectivesto develop and validate an electronic frailty index (eFI) using routinely available primary care electronic health record data.Study design and settingretrospective cohort study. Development and internal validation cohorts were established using a randomly split sample of the ResearchOne primary care database. External validation cohort established using THIN database.Participantspatients aged 65-95, registered with a ResearchOne or THIN practice on 14 October 2008.Predictorswe constructed the eFI using the cumulative deficit frailty model as our theoretical framework. The eFI score is calculated by the presence or absence of individual deficits as a proportion of the total possible. Categories of fit, mild, moderate and severe frailty were defined using population quartiles.Outcomesoutcomes were 1-, 3- and 5-year mortality, hospitalisation and nursing home admission.Statistical analysishazard ratios (HRs) were estimated using bivariate and multivariate Cox regression analyses. Discrimination was assessed using receiver operating characteristic (ROC) curves. Calibration was assessed using pseudo-R(2) estimates.Resultswe include data from a total of 931,541 patients. The eFI incorporates 36 deficits constructed using 2,171 CTV3 codes. One-year adjusted HR for mortality was 1.92 (95% CI 1.81-2.04) for mild frailty, 3.10 (95% CI 2.91-3.31) for moderate frailty and 4.52 (95% CI 4.16-4.91) for severe frailty. Corresponding estimates for hospitalisation were 1.93 (95% CI 1.86-2.01), 3.04 (95% CI 2.90-3.19) and 4.73 (95% CI 4.43-5.06) and for nursing home admission were 1.89 (95% CI 1.63-2.15), 3.19 (95% CI 2.73-3.73) and 4.76 (95% CI 3.92-5.77), with good to moderate discrimination but low calibration estimates.Conclusionsthe eFI uses routine data to identify older people with mild, moderate and severe frailty, with robust predictive validity for outcomes of mortality, hospitalisation and nursing home admission. Routine implementation of the eFI could enable delivery of evidence-based interventions to improve outcomes for this vulnerable group.

Project description:BackgroundHypertension is the most prevalent risk factor for mortality globally. Uncontrolled hypertension is associated with excess morbidity and mortality, and nearly one-half of individuals with hypertension do not have the condition under control. Data from electronic health record (EHR) systems may be useful for community hypertension surveillance, filling a gap in local public health departments' community health assessments and supporting the public health data modernization initiatives currently underway. To identify patients with hypertension, computable phenotypes are required. These phenotypes leverage available data elements-such as vitals measurements and medications-to identify patients diagnosed with hypertension. However, there are multiple methodologies for creating a phenotype, and the identification of which method most accurately reflects real-world prevalence rates is needed to support data modernization initiatives.ObjectiveThis study sought to assess the comparability of 6 different EHR-based hypertension prevalence estimates with estimates from a national survey. Each of the prevalence estimates was created using a different computable phenotype. The overarching goal is to identify which phenotypes most closely align with nationally accepted estimations.MethodsUsing the 6 different EHR-based computable phenotypes, we calculated hypertension prevalence estimates for Marion County, Indiana, for the period from 2014 to 2015. We extracted hypertension rates from the Behavioral Risk Factor Surveillance System (BRFSS) for the same period. We used the two 1-sided t test (TOST) to test equivalence between BRFSS- and EHR-based prevalence estimates. The TOST was performed at the overall level as well as stratified by age, gender, and race.ResultsUsing both 80% and 90% CIs, the TOST analysis resulted in 2 computable phenotypes demonstrating rough equivalence to BRFSS estimates. Variation in performance was noted across phenotypes as well as demographics. TOST with 80% CIs demonstrated that the phenotypes had less variance compared to BRFSS estimates within subpopulations, particularly those related to racial categories. Overall, less variance occurred on phenotypes that included vitals measurements.ConclusionsThis study demonstrates that certain EHR-derived prevalence estimates may serve as rough substitutes for population-based survey estimates. These outcomes demonstrate the importance of critically assessing which data elements to include in EHR-based computer phenotypes. Using comprehensive data sources, containing complete clinical data as well as data representative of the population, are crucial to producing robust estimates of chronic disease. As public health departments look toward data modernization activities, the EHR may serve to assist in more timely, locally representative estimates for chronic disease prevalence.

Project description:BackgroundTraditional methods for disease risk prediction and assessment, such as diagnostic tests using serum, urine, blood, saliva or imaging biomarkers, have been important for identifying high-risk individuals for many diseases, leading to early detection and improved survival. For pancreatic cancer, traditional methods for screening have been largely unsuccessful in identifying high-risk individuals in advance of disease progression leading to high mortality and poor survival. Electronic health records (EHR) linked to genetic profiles provide an opportunity to integrate multiple sources of patient information for risk prediction and stratification. We leverage a constellation of temporally associated diagnoses available in the EHR to construct a summary risk score, called a phenotype risk score (PheRS), for identifying individuals at high-risk for having pancreatic cancer. The proposed PheRS approach incorporates the time with respect to disease onset into the prediction framework. We combine and contrast the PheRS with more well-known measures of inherited susceptibility, namely, the polygenic risk scores (PRS) for prediction of pancreatic cancer.MethodologyWe first calculated pairwise, unadjusted associations between pancreatic cancer diagnosis and all possible other diagnoses across the medical phenome. We call these pairwise associations co-occurrences. After accounting for cross-phenotype correlations, the multivariable association estimates from a subset of relatively independent diagnoses were used to create a weighted sum PheRS. We constructed time-restricted risk scores using data from 38,359 participants in the Michigan Genomics Initiative (MGI) based on the diagnoses contained in the EHR at 0, 1, 2, and 5 years prior to the target pancreatic cancer diagnosis. The PheRS was assessed for predictability in the UK Biobank (UKB). We tested the relative contribution of PheRS when added to a model containing a summary measure of inherited genetic susceptibility (PRS) plus other covariates like age, sex, smoking status, drinking status, and body mass index (BMI).ResultsOur exploration of co-occurrence patterns identified expected associations while also revealing unexpected relationships that may warrant closer attention. Solely using the pancreatic cancer PheRS at 5 years before the target diagnoses yielded an AUC of 0.60 (95% CI = [0.58, 0.62]) in UKB. A larger predictive model including PheRS, PRS, and the covariates at the 5-year threshold achieved an AUC of 0.74 (95% CI = [0.72, 0.76]) in UKB. We note that PheRS does contribute independently in the joint model. Finally, scores at the top percentiles of the PheRS distribution demonstrated promise in terms of risk stratification. Scores in the top 2% were 10.20 (95% CI = [9.34, 12.99]) times more likely to identify cases than those in the bottom 98% in UKB at the 5-year threshold prior to pancreatic cancer diagnosis.ConclusionsWe developed a framework for creating a time-restricted PheRS from EHR data for pancreatic cancer using the rich information content of a medical phenome. In addition to identifying hypothesis-generating associations for future research, this PheRS demonstrates a potentially important contribution in identifying high-risk individuals, even after adjusting for PRS for pancreatic cancer and other traditional epidemiologic covariates. The methods are generalizable to other phenotypic traits.

Dataset Information

A strategy for validation of variables derived from large-scale electronic health record data.

Purpose

Methods

Results

Conclusions

Publications

A strategy for validation of variables derived from large-scale electronic health record data.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets