Dataset Information

Development of a Portable Tool to Identify Patients With Atrial Fibrillation Using Clinical Notes From the Electronic Medical Record.

ABSTRACT:

Background

The electronic medical record contains a wealth of information buried in free text. We created a natural language processing algorithm to identify patients with atrial fibrillation (AF) using text alone.

Methods and results

We created 3 data sets from patients with at least one AF billing code from 2010 to 2017: a training set (n=886), an internal validation set from site no. 1 (n=285), and an external validation set from site no. 2 (n=276). A team of clinicians reviewed and adjudicated patients as AF present or absent, which served as the reference standard. We trained 54 algorithms to classify each patient, varying the model, number of features, number of stop words, and the method used to create the feature set. The algorithm with the highest F-score (the harmonic mean of sensitivity and positive predictive value) in the training set was applied to the validation sets. F-scores and area under the receiver operating characteristic curves were compared between site no. 1 and site no. 2 using bootstrapping. Adjudicated AF prevalence was 75.1% at site no. 1 and 86.2% at site no. 2. Among 54 algorithms, the best performing model was logistic regression, using 1000 features, 100 stop words, and term frequency-inverse document frequency method to create the feature set, with sensitivity 92.8%, specificity 93.9%, and an area under the receiver operating characteristic curve of 0.93 in the training set. The performance at site no. 1 was sensitivity 92.5%, specificity 88.7%, with an area under the receiver operating characteristic curve of 0.91. The performance at site no. 2 was sensitivity 89.5%, specificity 71.1%, with an area under the receiver operating characteristic curve of 0.80. The F-score was lower at site no. 2 compared with site no. 1 (92.5% [SD, 1.1%] versus 94.2% [SD, 1.1%]; P<0.001).

Conclusions

We developed a natural language processing algorithm to identify patients with AF using text alone, with >90% F-score at 2 separate sites. This approach allows better use of the clinical narrative and creates an opportunity for precise, high-throughput cohort identification.

SUBMITTER: Shah RU

PROVIDER: S-EPMC7646941 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundAtrial fibrillation (AF) burden on patients and healthcare systems warrants innovative strategies for screening asymptomatic individuals.ObjectiveWe sought to externally validate a predictive model originally developed in a German population to detect unidentified incident AF utilising real-world primary healthcare databases from countries in Europe and Australia.MethodsThis retrospective cohort study used anonymized, longitudinal patient data from 5 country-level primary care databases, including Australia, Belgium, France, Germany, and the UK. The study eligibility included adult patients (≥45 years) with either an AF diagnosis (cases) or no diagnosis (controls) who had continuous enrolment in the respective database prior to the study period. Logistic regression was fitted to a binary response (yes/no) for AF diagnosis using pre-determined risk factors.ResultsAF patients were from Germany (n = 63,562), the UK (n = 42,652), France (n = 7,213), Australia (n = 2,753), and Belgium (n = 1,371). Cases were more likely to have hypertension or other cardiac conditions than controls in all validation datasets compared to the model development data. The area under the receiver operating characteristic (ROC) curve in the validation datasets ranged from 0.79 (Belgium) to 0.84 (Germany), comparable to the German study model, which had an area under the curve of 0.83. Most validation sets reported similar specificity at approximately 80% sensitivity, ranging from 67% (France) to 71% (United Kingdom). The positive predictive value (PPV) ranged from 2% (Belgium) to 16% (Germany), and the number needed to be screened was 50 in Belgium and 6 in Germany. The prevalence of AF varied widely between these datasets, which may be related to different coding practices. Low prevalence affected PPV, but not sensitivity, specificity, and ROC curves.ConclusionsAF risk prediction algorithms offer targeted ways to identify patients using electronic health records, which could improve screening number and the cost-effectiveness of AF screening if implemented in clinical practice.

Project description:BACKGROUND:Many health care systems now allow patients to access their electronic health record (EHR) notes online through patient portals. Medical jargon in EHR notes can confuse patients, which may interfere with potential benefits of patient access to EHR notes. OBJECTIVE:The aim of this study was to develop and evaluate the usability and content quality of NoteAid, a Web-based natural language processing system that links medical terms in EHR notes to lay definitions, that is, definitions easily understood by lay people. METHODS:NoteAid incorporates two core components: CoDeMed, a lexical resource of lay definitions for medical terms, and MedLink, a computational unit that links medical terms to lay definitions. We developed innovative computational methods, including an adapted distant supervision algorithm to prioritize medical terms important for EHR comprehension to facilitate the effort of building CoDeMed. Ten physician domain experts evaluated the user interface and content quality of NoteAid. The evaluation protocol included a cognitive walkthrough session and a postsession questionnaire. Physician feedback sessions were audio-recorded. We used standard content analysis methods to analyze qualitative data from these sessions. RESULTS:Physician feedback was mixed. Positive feedback on NoteAid included (1) Easy to use, (2) Good visual display, (3) Satisfactory system speed, and (4) Adequate lay definitions. Opportunities for improvement arising from evaluation sessions and feedback included (1) improving the display of definitions for partially matched terms, (2) including more medical terms in CoDeMed, (3) improving the handling of terms whose definitions vary depending on different contexts, and (4) standardizing the scope of definitions for medicines. On the basis of these results, we have improved NoteAid's user interface and a number of definitions, and added 4502 more definitions in CoDeMed. CONCLUSIONS:Physician evaluation yielded useful feedback for content validation and refinement of this innovative tool that has the potential to improve patient EHR comprehension and experience using patient portals. Future ongoing work will develop algorithms to handle ambiguous medical terms and test and evaluate NoteAid with patients.

Project description:Importance:Atrial fibrillation (AF) contributes to substantial morbidity, mortality, and health care expenditures. Accurate prediction of incident AF would enhance AF management and potentially improve patient outcomes. Objective:To validate the AF risk prediction model originally developed by the Cohorts for Heart and Aging Research in Genomic Epidemiology-Atrial Fibrillation (CHARGE-AF) investigators using a large repository of electronic medical records (EMRs). Design, Setting, and Participants:In this prediction model study, deidentified EMRs of 33?494 individuals 40 years or older who were white or African American and had no history of AF were reviewed and analyzed. The participants were followed up in the internal medicine outpatient clinics at Vanderbilt University Medical Center for incident AF from December 31, 2005, until December 31, 2010. Adjusting for differences in baseline hazard, the CHARGE-AF Cox proportional hazards model regression coefficients were applied to the EMR cohort. A simple version of the model with no echocardiographic variables was also evaluated. Data were analyzed from October 31, 2013, to January 31, 2014. Main Outcomes and Measures:Incident AF. Predictors in the model included age, race, height, weight, systolic and diastolic blood pressure, treatment for hypertension, smoking status, type 2 diabetes, heart failure, history of myocardial infarction, left ventricular hypertrophy, and PR interval. Results:Among the 33?494 participants, the median age was 57 (interquartile range, 49-67) years; 57% of patients were women, 43% were men, 85.7% were white, and 14.3% were African American. During the mean (SD) follow-up of 4.8 (0.9) years, 2455 individuals (7.3%) developed AF. Both models had poor calibration in the EMR cohort, with underprediction of AF among low-risk individuals and overprediction of AF among high-risk individuals (10th and 90th percentiles for predicted probability of incident AF, 0.005 and 0.179, respectively). The full CHARGE-AF model had a C index of 0.708 (95% CI, 0.699-0.718) in our cohort. The simple model had similar discrimination (C index, 0.709; 95% CI, 0.699-0.718; P?=?.70 for difference between models). Conclusions and Relevance:Despite reasonable discrimination, the CHARGE-AF models showed poor calibration in this EMR cohort. This study highlights the difficulties of applying a risk model derived from prospective cohort studies to an EMR cohort and suggests that these AF risk prediction models be used with caution in the EMR setting. Future risk models may need to be developed and validated within EMR cohorts.