Dataset Information

Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods.

ABSTRACT: OBJECTIVE:To systematically review published literature and identify consistency and variation in the aims, measures, and methods of studies using electronic health record (EHR) audit logs to observe clinical activities. MATERIALS AND METHODS:In July 2019, we searched PubMed for articles using EHR audit logs to study clinical activities. We coded and clustered the aims, measures, and methods of each article into recurring categories. We likewise extracted and summarized the methods used to validate measures derived from audit logs and limitations discussed of using audit logs for research. RESULTS:Eighty-five articles met inclusion criteria. Study aims included examining EHR use, care team dynamics, and clinical workflows. Studies employed 6 key audit log measures: counts of actions captured by audit logs (eg, problem list viewed), counts of higher-level activities imputed by researchers (eg, chart review), activity durations, activity sequences, activity clusters, and EHR user networks. Methods used to preprocess audit logs varied, including how authors filtered extraneous actions, mapped actions to higher-level activities, and interpreted repeated actions or gaps in activity. Nineteen studies validated results (22%), but only 9 (11%) through direct observation, demonstrating varying levels of measure accuracy. DISCUSSION:While originally designed to aid access control, EHR audit logs have been used to observe diverse clinical activities. However, most studies lack sufficient discussion of measure definition, calculation, and validation to support replication, comparison, and cross-study synthesis. CONCLUSION:EHR audit logs have potential to scale observational research but the complexity of audit log measures necessitates greater methodological transparency and validated standards.

SUBMITTER: Rule A

PROVIDER: S-EPMC7025338 | biostudies-literature | 2020 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods.

Rule Adam A Chiang Michael F MF Hribar Michelle R MR

Journal of the American Medical Informatics Association : JAMIA 20200301 3

<h4>Objective</h4>To systematically review published literature and identify consistency and variation in the aims, measures, and methods of studies using electronic health record (EHR) audit logs to observe clinical activities.<h4>Materials and methods</h4>In July 2019, we searched PubMed for articles using EHR audit logs to study clinical activities. We coded and clustered the aims, measures, and methods of each article into recurring categories. We likewise extracted and summarized the method ...[more]

PMID: 31750912

Similar Datasets

Project description:BACKGROUND:Electronic health record (EHR) access and audit logs record behaviors of providers as they navigate the EHR. These data can be used to better understand provider responses to EHR-based clinical decision support (CDS), shedding light on whether and why CDS is effective. OBJECTIVE:This study aimed to determine the feasibility of using EHR access and audit logs to track primary care physicians' (PCPs') opening of and response to noninterruptive alerts delivered to EHR InBaskets. METHODS:We conducted a descriptive study to assess the use of EHR log data to track provider behavior. We analyzed data recorded following opening of 799 noninterruptive alerts sent to 75 PCPs' InBaskets through a prior randomized controlled trial. Three types of alerts highlighted new medication concerns for older patients' posthospital discharge: information only (n=593), medication recommendations (n=37), and test recommendations (n=169). We sought log data to identify the person opening the alert and the timing and type of PCPs' follow-up EHR actions (immediate vs by the end of the following day). We performed multivariate analyses examining associations between alert type, patient characteristics, provider characteristics, and contextual factors and likelihood of immediate or subsequent PCP action (general, medication-specific, or laboratory-specific actions). We describe challenges and strategies for log data use. RESULTS:We successfully identified the required data in EHR access and audit logs. More than three-quarters of alerts (78.5%, 627/799) were opened by the PCP to whom they were directed, allowing us to assess immediate PCP action; of these, 208 alerts were followed by immediate action. Expanding on our analyses to include alerts opened by staff or covering physicians, we found that an additional 330 of the 799 alerts demonstrated PCP action by the end of the following day. The remaining 261 alerts showed no PCP action. Compared to information-only alerts, the odds ratio (OR) of immediate action was 4.03 (95% CI 1.67-9.72) for medication-recommendation and 2.14 (95% CI 1.38-3.32) for test-recommendation alerts. Compared to information-only alerts, ORs of medication-specific action by end of the following day were significantly greater for medication recommendations (5.59; 95% CI 2.42-12.94) and test recommendations (1.71; 95% CI 1.09-2.68). We found a similar pattern for OR of laboratory-specific action. We encountered 2 main challenges: (1) Capturing a historical snapshot of EHR status (number of InBasket messages at time of alert delivery) required incorporation of data generated many months prior with longitudinal follow-up. (2) Accurately interpreting data elements required iterative work by a physician/data manager team taking action within the EHR and then examining audit logs to identify corresponding documentation. CONCLUSIONS:EHR log data could inform future efforts and provide valuable information during development and refinement of CDS interventions. To address challenges, use of these data should be planned before implementing an EHR-based study.

Project description:AimsThe diagnosis of heart failure (HF) is an important problem in primary care. We previously demonstrated a 74% increase in registered HF diagnoses in primary care electronic health records (EHRs) following an extended audit procedure. What remains unclear is the accuracy of registered HF pre-audit and which EHR variables are most important in the extended audit strategy. This study aims to describe the diagnostic HF classification sequence at different stages, assess general practitioner (GP) HF misclassification, and test the predictive performance of an optimized audit.Methods and resultsThis is a secondary analysis of the OSCAR-HF study, a prospective observational trial including 51 participating GPs. OSCAR used an extended audit based on typical HF risk factors, signs, symptoms, and medications in GPs' EHR. This resulted in a list of possible HF patients, which participating GPs had to classify as HF or non-HF. We compared registered HF diagnoses before and after GPs' assessment. For our analysis of audit performance, we used GPs' assessment of HF as primary outcome and audit queries as dichotomous predictor variables for a gradient boosted machine (GBM) decision tree algorithm and logistic regression model. Of the 18 011 patients eligible for the audit intervention, 4678 (26.0%) were identified as possible HF patients and submitted for GPs' assessment in the audit stage. There were 310 patients with registered HF before GP assessment, of whom 146 (47.1%) were judged not to have HF by their GP (over-registration). There were 538 patients with registered HF after GP assessment, of whom 374 (69.5%) did not have registered HF before GP assessment (under-registration). The GBM and logistic regression model had a comparable predictive performance (area under the curve of 0.70 [95% confidence interval 0.65-0.77] and 0.69 [95% confidence interval 0.64-0.75], respectively). This was not significantly impacted by reducing the set of predictor variables to the 10 most important variables identified in the GBM model (free-text and coded cardiomyopathy, ischaemic heart disease and atrial fibrillation, digoxin, mineralocorticoid receptor antagonists, and combinations of renin-angiotensin system inhibitors and beta-blockers with diuretics). This optimized query set was enough to identify 86% (n = 461/538) of GPs' self-assessed HF population with a 33% reduction (n = 1537/4678) in screening caseload.ConclusionsDiagnostic coding of HF in primary care health records is inaccurate with a high degree of under-registration and over-registration. An optimized query set enabled identification of more than 80% of GPs' self-assessed HF population.

Project description:BackgroundHypertension is the most prevalent risk factor for mortality globally. Uncontrolled hypertension is associated with excess morbidity and mortality, and nearly one-half of individuals with hypertension do not have the condition under control. Data from electronic health record (EHR) systems may be useful for community hypertension surveillance, filling a gap in local public health departments' community health assessments and supporting the public health data modernization initiatives currently underway. To identify patients with hypertension, computable phenotypes are required. These phenotypes leverage available data elements-such as vitals measurements and medications-to identify patients diagnosed with hypertension. However, there are multiple methodologies for creating a phenotype, and the identification of which method most accurately reflects real-world prevalence rates is needed to support data modernization initiatives.ObjectiveThis study sought to assess the comparability of 6 different EHR-based hypertension prevalence estimates with estimates from a national survey. Each of the prevalence estimates was created using a different computable phenotype. The overarching goal is to identify which phenotypes most closely align with nationally accepted estimations.MethodsUsing the 6 different EHR-based computable phenotypes, we calculated hypertension prevalence estimates for Marion County, Indiana, for the period from 2014 to 2015. We extracted hypertension rates from the Behavioral Risk Factor Surveillance System (BRFSS) for the same period. We used the two 1-sided t test (TOST) to test equivalence between BRFSS- and EHR-based prevalence estimates. The TOST was performed at the overall level as well as stratified by age, gender, and race.ResultsUsing both 80% and 90% CIs, the TOST analysis resulted in 2 computable phenotypes demonstrating rough equivalence to BRFSS estimates. Variation in performance was noted across phenotypes as well as demographics. TOST with 80% CIs demonstrated that the phenotypes had less variance compared to BRFSS estimates within subpopulations, particularly those related to racial categories. Overall, less variance occurred on phenotypes that included vitals measurements.ConclusionsThis study demonstrates that certain EHR-derived prevalence estimates may serve as rough substitutes for population-based survey estimates. These outcomes demonstrate the importance of critically assessing which data elements to include in EHR-based computer phenotypes. Using comprehensive data sources, containing complete clinical data as well as data representative of the population, are crucial to producing robust estimates of chronic disease. As public health departments look toward data modernization activities, the EHR may serve to assist in more timely, locally representative estimates for chronic disease prevalence.

Project description:BackgroundThe Meaningful Use (MU) program has increased the national emphasis on electronic measurement of hospital quality.ObjectiveTo evaluate stroke MU and one VHA stroke electronic clinical quality measure (eCQM) in national VHA data and determine sources of error in using centralized electronic health record (EHR) data.DesignOur study is a retrospective cross-sectional study of stroke quality measure eCQMs vs. chart review in a national EHR. We developed local SQL algorithms to generate the eCQMs, then modified them to run on VHA Central Data Warehouse (CDW) data. eCQM results were generated from CDW data in 2130 ischemic stroke admissions in 11 VHA hospitals. Local and CDW results were compared to chart review.Main measuresWe calculated the raw proportion of matching cases, sensitivity/specificity, and positive/negative predictive values (PPV/NPV) for the numerators and denominators of each eCQM. To assess overall agreement for each eCQM, we calculated a weighted kappa and prevalence-adjusted bias-adjusted kappa statistic for a three-level outcome: ineligible, eligible-passed, or eligible-failed.Key resultsIn five eCQMs, the proportion of matched cases between CDW and chart ranged from 95.4 %-99.7 % (denominators) and 87.7 %-97.9 % (numerators). PPVs tended to be higher (range 96.8 %-100 % in CDW) with NPVs less stable and lower. Prevalence-adjusted bias-adjusted kappas for overall agreement ranged from 0.73-0.95. Common errors included difficulty in identifying: (1) mechanical VTE prophylaxis devices, (2) hospice and other specific discharge disposition, and (3) contraindications to receiving care processes.ConclusionsStroke MU indicators can be relatively accurately generated from existing EHR systems (nearly 90 % match to chart review), but accuracy decreases slightly in central compared to local data sources. To improve stroke MU measure accuracy, EHRs should include standardized data elements for devices, discharge disposition (including hospice and comfort care status), and recording contraindications.

Dataset Information

Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods.

Publications

Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets