Dataset Information

ComprehENotes, an Instrument to Assess Patient Reading Comprehension of Electronic Health Record Notes: Development and Validation.

ABSTRACT: BACKGROUND:Patient portals are widely adopted in the United States and allow millions of patients access to their electronic health records (EHRs), including their EHR clinical notes. A patient's ability to understand the information in the EHR is dependent on their overall health literacy. Although many tests of health literacy exist, none specifically focuses on EHR note comprehension. OBJECTIVE:The aim of this paper was to develop an instrument to assess patients' EHR note comprehension. METHODS:We identified 6 common diseases or conditions (heart failure, diabetes, cancer, hypertension, chronic obstructive pulmonary disease, and liver failure) and selected 5 representative EHR notes for each disease or condition. One note that did not contain natural language text was removed. Questions were generated from these notes using Sentence Verification Technique and were analyzed using item response theory (IRT) to identify a set of questions that represent a good test of ability for EHR note comprehension. RESULTS:Using Sentence Verification Technique, 154 questions were generated from the 29 EHR notes initially obtained. Of these, 83 were manually selected for inclusion in the Amazon Mechanical Turk crowdsourcing tasks and 55 were ultimately retained following IRT analysis. A follow-up validation with a second Amazon Mechanical Turk task and IRT analysis confirmed that the 55 questions test a latent ability dimension for EHR note comprehension. A short test of 14 items was created along with the 55-item test. CONCLUSIONS:We developed ComprehENotes, an instrument for assessing EHR note comprehension from existing EHR notes, gathered responses using crowdsourcing, and used IRT to analyze those responses, thus resulting in a set of questions to measure EHR note comprehension. Crowdsourced responses from Amazon Mechanical Turk can be used to estimate item parameters and select a subset of items for inclusion in the test set using IRT. The final set of questions is the first test of EHR note comprehension.

SUBMITTER: Lalor JP

PROVIDER: S-EPMC5943623 | biostudies-literature | 2018 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

ComprehENotes, an Instrument to Assess Patient Reading Comprehension of Electronic Health Record Notes: Development and Validation.

Lalor John P JP Wu Hao H Chen Li L Mazor Kathleen M KM Yu Hong H

Journal of medical Internet research 20180425 4

<h4>Background</h4>Patient portals are widely adopted in the United States and allow millions of patients access to their electronic health records (EHRs), including their EHR clinical notes. A patient's ability to understand the information in the EHR is dependent on their overall health literacy. Although many tests of health literacy exist, none specifically focuses on EHR note comprehension.<h4>Objective</h4>The aim of this paper was to develop an instrument to assess patients' EHR note comp ...[more]

PMID: 29695372

Similar Datasets

Project description:BackgroundMedical terms are a major obstacle for patients to comprehend their electronic health record (EHR) notes. Clinical natural language processing (NLP) systems that link EHR terms to lay terms or definitions allow patients to easily access helpful information when reading through their EHR notes, and have shown to improve patient EHR comprehension. However, high-quality lay language resources for EHR terms are very limited in the public domain. Because expanding and curating such a resource is a costly process, it is beneficial and even necessary to identify terms important for patient EHR comprehension first.ObjectiveWe aimed to develop an NLP system, called adapted distant supervision (ADS), to rank candidate terms mined from EHR corpora. We will give EHR terms ranked as high by ADS a higher priority for lay language annotation-that is, creating lay definitions for these terms.MethodsAdapted distant supervision uses distant supervision from consumer health vocabulary and transfer learning to adapt itself to solve the problem of ranking EHR terms in the target domain. We investigated 2 state-of-the-art transfer learning algorithms (ie, feature space augmentation and supervised distant supervision) and designed 5 types of learning features, including distributed word representations learned from large EHR data for ADS. For evaluating ADS, we asked domain experts to annotate 6038 candidate terms as important or nonimportant for EHR comprehension. We then randomly divided these data into the target-domain training data (1000 examples) and the evaluation data (5038 examples). We compared ADS with 2 strong baselines, including standard supervised learning, on the evaluation data.ResultsThe ADS system using feature space augmentation achieved the best average precision, 0.850, on the evaluation set when using 1000 target-domain training examples. The ADS system using supervised distant supervision achieved the best average precision, 0.819, on the evaluation set when using only 100 target-domain training examples. The 2 ADS systems both performed significantly better than the baseline systems (P<.001 for all measures and all conditions). Using a rich set of learning features contributed to ADS's performance substantially.ConclusionsADS can effectively rank terms mined from EHRs. Transfer learning improved ADS's performance even with a small number of target-domain training examples. EHR terms prioritized by ADS were used to expand a lay language resource that supports patient EHR comprehension. The top 10,000 EHR terms ranked by ADS are available upon request.

Project description:ObjectivesTo develop and evaluate the validity of a scale to assess patients' perceived benefits and risks of reading ambulatory visit notes online (open notes).DesignFour studies were used to evaluate the construct validity of a benefits and risks scale. Study 1 refined the items; study 2 evaluated underlying factor structure and identified the items; study 3 evaluated study 2 results in a separate sample; and study 4 examined factorial invariance of the developed scale across educational subsamples.SettingAmbulatory care in three large health systems in the USA.ParticipantsParticipants in three US health systems who responded to one of two online surveys asking about benefits and risks of reading visit notes: a psychometrics survey of primary care patients, and a large general survey of patients across all ambulatory specialties. Sample sizes: n=439 (study 1); n=439 (study 2); n=500 (study 3); and n=250 (study 4).Primary and secondary outcome measuresQuestionnaire items about patients' perceived benefits and risks of reading online visit notes.ResultsStudy 1 resulted in the selection of a 10-point importance response option format over a 4-point agreement scale. Exploratory factor analysis (EFA) in study 2 resulted in two-factor solution: a four-item benefits factor with good reliability (alpha=0.83) and a three-item risks factor with poor reliability (alpha=0.52). The factor structure was confirmed in study 3, and confirmatory factor analysis of benefit items resulted in an excellent fitting model, X2(2)=2.949; confirmatory factor index=0.998; root mean square error of approximation=0.04 (0.00, 0.142); loadings 0.68-0.86; alpha=0.88. Study 4 supported configural, measurement and structural invariance for the benefits scale across high and low-education patient groups.ConclusionsThe findings suggest that the four-item benefits scale has excellent construct validity and preliminary evidence of generalising across different patient populations. Further scale development is needed to understand perceived risks of reading open notes.

Project description:BackgroundIt is assumed that the implementation of health information technology introduces new vulnerabilities within a complex sociotechnical health care system, but no international consensus exists on a standardized format for enhancing the collection, analysis, and interpretation of technology-induced errors.ObjectiveThis study aims to develop a classification for patient safety incident reporting associated with the use of mature electronic health records (EHRs). It also aims to validate the classification by using a data set of incidents during a 6-month period immediately after the implementation of a new EHR system.MethodsThe starting point of the classification development was the Finnish Technology-Induced Error Risk Assessment Scale tool, based on research on commonly recognized error types. A multiprofessional research team used iterative tests on consensus building to develop a classification system. The final classification, with preliminary descriptions of classes, was validated by applying it to analyze EHR-related error incidents (n=428) during the implementation phase of a new EHR system and also to evaluate this classification's characteristics and applicability for reporting incidents. Interrater agreement was applied.ResultsThe number of EHR-related patient safety incidents during the implementation period (n=501) was five-fold when compared with the preimplementation period (n=82). The literature identified new error types that were added to the emerging classification. Error types were adapted iteratively after several test rounds to develop a classification for reporting patient safety incidents in the clinical use of a high-maturity EHR system. Of the 427 classified patient safety incidents, interface problems accounted for 96 (22.5%) incident reports, usability problems for 73 (17.1%), documentation problems for 60 (14.1%), and clinical workflow problems for 33 (7.7%). Altogether, 20.8% (89/427) of reports were related to medication section problems, and downtime problems were rare (n=8). During the classification work, 14.8% (74/501) of reports of the original sample were rejected because of insufficient information, even though the reports were deemed to be related to EHRs. The interrater agreement during the blinded review was 97.7%.ConclusionsThis study presents a new classification for EHR-related patient safety incidents applicable to mature EHRs. The number of EHR-related patient safety incidents during the implementation period may reflect patient safety challenges during the implementation of a new type of high-maturity EHR system. The results indicate that the types of errors previously identified in the literature change with the EHR development cycle.

Project description:ImportanceAs health information transparency increases, patients more often seek their health data. More than 44 million patients in the US can now readily access their ambulatory visit notes online, and the practice is increasing abroad. Few studies have assessed documentation errors that patients identify in their notes and how these may inform patient engagement and safety strategies.ObjectiveTo assess the frequency and types of errors identified by patients who read open ambulatory visit notes.Design, setting, and participantsIn this survey study, a total of 136?815 patients at 3 US health care organizations with open notes, including 79 academic and community ambulatory care practices, received invitations to an online survey from June 5 to October 20, 2017. Patients who had at least 1 ambulatory note and had logged onto the portal at least once in the past 12 months were included. Data analysis was performed from July 3, 2018, to April 27, 2020.ExposuresAccess to ambulatory care open notes through patient portals for up to 7 years (2010-2017).Main outcomes and measuresProportion of patients reporting a mistake and how serious they perceived the mistake to be, factors associated with finding errors characterized by patients as serious, and categories of patient-reported errors.ResultsOf 136?815 patients who received survey invitations, 29?656 (21.7%) responded and 22?889 patients (mean [SD] age, 55.16 [15.96] years; 14 447 [63.1%] female; 18 301 [80.0%] white) read 1 or more notes in the past 12 months and completed error questions. Of these patients, 4830 (21.1%) reported a perceived mistake and 2043 (42.3%) reported that the mistake was serious (somewhat serious: 1563 [32.4%]; very serious: 480 [9.9%]). In multivariable analysis, female patients (relative risk [RR], 1.79; 95% CI, 1.72-1.85), more educated patients (RR, 1.38; 95% CI, 1.29-1.48), sicker patients (RR, 1.89; 95% CI, 1.84-1.94), those aged 45 to 64 years (RR, 2.23; 95% CI, 2.06-2.42), those 65 years or older (RR, 2.00; 95% CI, 1.73-2.32), and those who read more than 1 note (2-3 notes: RR, 1.82; 95% CI, 1.34-2.47; ?4 notes: RR, 3.09; 95% CI, 2.02-4.73) were more likely to report a mistake that they found to be serious compared with their reference groups. After categorization of patient-reported very serious mistakes, those specifically mentioning the word diagnosis or describing a specific error in current or past diagnoses were most common (98 of 356 [27.5%]), followed by inaccurate medical history (85 of 356 [23.9%]), medications or allergies (50 of 356 [14.0%]), and tests, procedures, or results (30 of 356 [8.4%]). A total of 23 (6.5%) reflected notes reportedly written on the wrong patient. Of 433 very serious errors, 255 (58.9%) included at least 1 perceived error potentially associated with the diagnostic process (eg, history, physical examination, tests, referrals, and communication).Conclusions and relevanceIn this study, patients who read ambulatory notes online perceived mistakes, a substantial proportion of which they found to be serious. Older and sicker patients were twice as likely to report a serious error compared with younger and healthier patients, indicating important safety and quality implications. Sharing notes with patients may help engage them to improve record accuracy and health care safety together with practitioners.

Project description:ObjectiveTo adapt and validate a questionnaire originally developed in a research setting for assessment of comprehension of consent information in a different cultural and linguistic research setting.DesignThe adaptation process involved development and customisation of a questionnaire for each of the three study groups, modelled closely on the previously validated questionnaire. The three adapted draft questionnaires were further reviewed by two bioethicists and the developer of the original questionnaire for face and content validity. The revised questionnaire was subsequently programmed into an audio computerised format, with translations and back translations in three widely spoken languages by the study participants: Luo, Swahili and English.SettingThe questionnaire was validated among adolescents, their parents and young adults living in Siaya County, a rural region of western Kenya.ParticipantsTwenty-five-item adapted questionnaires consisting of close-ended, multiple-choice and open-ended questions were administered to 235 participants consisting of 107 adolescents, 92 parents and 36 young adults. Test-retest was conducted 2-4 weeks after first questionnaire administration among 74 adolescents, young adults and parents.Outcome measurePrimary outcome measures included ceiling/floor analysis to identify questions with extremes in responses and item-level correlation to determine the test-retest relationships. Given the data format, tetrachoric correlations were conducted for dichotomous items and polychoric correlations for ordinal items. The qualitative validation assessment included face and content validity evaluation of the adapted instrument by technical experts.ResultsCeiling/floor analysis showed eight question items for which >80% of one or more groups responded correctly, while for nine questions, including all seven open-ended questions,<20% responded correctly. Majority of the question items had moderate to strong test-retest correlation estimates indicating temporal stability.ConclusionsOur study demonstrates that cross-cultural adaptation and validation of an informed consent comprehension questionnaire is feasible. However, further research is needed to develop a tool which can estimate a quantifiable threshold of comprehension thereby serving as an objective indicator of the need for interventions to improve comprehension.

Dataset Information

ComprehENotes, an Instrument to Assess Patient Reading Comprehension of Electronic Health Record Notes: Development and Validation.

Publications

ComprehENotes, an Instrument to Assess Patient Reading Comprehension of Electronic Health Record Notes: Development and Validation.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets