Dataset Information

Cranky comments: detecting clinical decision support malfunctions through free-text override reasons.

ABSTRACT:

Background

Rule-base clinical decision support alerts are known to malfunction, but tools for discovering malfunctions are limited.

Objective

Investigate whether user override comments can be used to discover malfunctions.

Methods

We manually classified all rules in our database with at least 10 override comments into 3 categories based on a sample of override comments: "broken," "not broken, but could be improved," and "not broken." We used 3 methods (frequency of comments, cranky word list heuristic, and a Naïve Bayes classifier trained on a sample of comments) to automatically rank rules based on features of their override comments. We evaluated each ranking using the manual classification as truth.

Results

Of the rules investigated, 62 were broken, 13 could be improved, and the remaining 45 were not broken. Frequency of comments performed worse than a random ranking, with precision at 20 of 8 and AUC = 0.487. The cranky comments heuristic performed better with precision at 20 of 16 and AUC = 0.723. The Naïve Bayes classifier had precision at 20 of 17 and AUC = 0.738.

Discussion

Override comments uncovered malfunctions in 26% of all rules active in our system. This is a lower bound on total malfunctions and much higher than expected. Even for low-resource organizations, reviewing comments identified by the cranky word list heuristic may be an effective and feasible way of finding broken alerts.

Conclusion

Override comments are a rich data source for finding alerts that are broken or could be improved. If possible, we recommend monitoring all override comments on a regular basis.

SUBMITTER: Aaron S

PROVIDER: S-EPMC6308015 | biostudies-literature | 2019 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Cranky comments: detecting clinical decision support malfunctions through free-text override reasons.

Aaron Skye S McEvoy Dustin S DS Ray Soumi S Hickman Thu-Trang T TT Wright Adam A

Journal of the American Medical Informatics Association : JAMIA 20190101 1

<h4>Background</h4>Rule-base clinical decision support alerts are known to malfunction, but tools for discovering malfunctions are limited.<h4>Objective</h4>Investigate whether user override comments can be used to discover malfunctions.<h4>Methods</h4>We manually classified all rules in our database with at least 10 override comments into 3 categories based on a sample of override comments: "broken," "not broken, but could be improved," and "not broken." We used 3 methods (frequency of comments ...[more]

PMID: 30590557

Similar Datasets

Project description:BackgroundDocumentation burden is a common problem with modern electronic health record (EHR) systems. To reduce this burden, various recording methods (eg, voice recorders or motion sensors) have been proposed. However, these solutions are in an early prototype phase and are unlikely to transition into practice in the near future. A more pragmatic alternative is to directly modify the implementation of the existing functionalities of an EHR system.ObjectiveThis study aims to assess the nature of free-text comments entered into EHR flowsheets that supplement quantitative vital sign values and examine opportunities to simplify functionality and reduce documentation burden.MethodsWe evaluated 209,055 vital sign comments in flowsheets that were generated in the Epic EHR system at the Vanderbilt University Medical Center in 2018. We applied topic modeling, as well as the natural language processing Clinical Language Annotation, Modeling, and Processing software system, to extract generally discussed topics and detailed medical terms (expressed as probability distribution) to investigate the stories communicated in these comments.ResultsOur analysis showed that 63.33% (6053/9557) of the users who entered vital signs made at least one free-text comment in vital sign flowsheet entries. The user roles that were most likely to compose comments were registered nurse, technician, and licensed nurse. The most frequently identified topics were the notification of a result to health care providers (0.347), the context of a measurement (0.307), and an inability to obtain a vital sign (0.224). There were 4187 unique medical terms that were extracted from 46,029 (0.220) comments, including many symptom-related terms such as "pain," "upset," "dizziness," "coughing," "anxiety," "distress," and "fever" and drug-related terms such as "tylenol," "anesthesia," "cannula," "oxygen," "motrin," "rituxan," and "labetalol."ConclusionsConsidering that flowsheet comments are generally not displayed or automatically pulled into any clinical notes, our findings suggest that the flowsheet comment functionality can be simplified (eg, via structured response fields instead of a text input dialog) to reduce health care provider effort. Moreover, rich and clinically important medical terms such as medications and symptoms should be explicitly recorded in clinical notes for better visibility.

Project description:BackgroundPatient-reported experience surveys allow administrators, clinicians, and researchers to quantify and improve health care by receiving feedback directly from patients. Existing research has focused primarily on quantitative analysis of survey items, but these measures may collect optional free-text comments. These comments can provide insights for health systems but may not be analyzed due to limited resources and the complexity of traditional textual analysis. However, advances in machine learning-based natural language processing provide opportunities to learn from this traditionally underused data source.ObjectiveThis study aimed to apply natural language processing to model topics found in free-text comments of patient-reported experience surveys.MethodsConsumer Assessment of Healthcare Providers and Systems-derived patient experience surveys were collected and linked to administrative inpatient records by the provincial health services organization responsible for inpatient care. Unsupervised topic modeling with automated labeling was performed with BERTopic. Sentiment analysis was performed to further assist in topic description.ResultsBetween April 2016 and February 2020, 43.4% (43,522/100,272) adult patients and 46.9% (3501/7464) pediatric caregivers included free-text responses on completed patient experience surveys. Topic models identified 86 topics among adult survey responses and 35 topics among pediatric responses that included elements of care not currently surveyed by existing questionnaires. Frequent topics were generally positive.ConclusionsWe found that with limited tuning, BERTopic identified care experience topics with interpretable automated labeling. Results are discussed in the context of person-centered care, patient safety, and health care quality improvement. Furthermore, we note the opportunity for the identification of temporal and site-specific trends as a method to identify patient care and safety concerns. As the use of patient experience measurement increases in health care, we discuss how machine learning can be leveraged to provide additional insight on patient experiences.

Project description:ObjectivesTo analyse free-text responses from the first Scottish Cancer Patient Experience Survey (SCPES) to understand patients' experiences of care, identify valued aspects and areas for improvement.DesignInductive thematic analysis of seven free-text comment boxes covering all stages of the cancer experience, from a national cohort survey.Setting and participantsAdult cancer patients diagnosed across all Health Boards in Scotland between July 2013 and March 2014, and who had an inpatient stay or hospital visit between January and September 2014. 2663 respondents (of n=4835 survey respondents) provided 6961 free-text comments.Main outcome measuresPositive and negative themes of patients' experiences. Differences in the proportion of positive to negative comments by demographics.MethodsData were analysed as follows: (1) comments were initially categorised at a high level (eg, positive, negative, miscellaneous, etc); (2) inductive codes were derived and applied to all relevant comments; (3) codes sharing similar meaning were amalgamated into subthemes, and code frequencies were measured; (4) subthemes were mapped into overarching themes; (5) difference in the proportion of positive to negative comments by demographics were analysed using χ2 tests.ResultsParticipants made more positive than negative comments (1:0.78). Analysis highlighted the importance to patients of Feeling that Individual Needs Are Met and Feeling Confident Within the System. Comments also provided insight into how Processes and Structures within the system of care can negatively impact on patients' experience. Particular issues were identified with care experiences in the lead-up to diagnosis.ConclusionsThis analysis provides a detailed understanding of patients' cancer care experiences, therefore indicating what aspects matter in those experiences. Although the majority of comments were positive, there were a significant number of negative comments, especially about the lead-up to diagnosis. Comments suggest patients would value greater integration of care from services involved in their treatment for cancer.

Dataset Information

Cranky comments: detecting clinical decision support malfunctions through free-text override reasons.

Background

Objective

Methods

Results

Discussion

Conclusion

Publications

Cranky comments: detecting clinical decision support malfunctions through free-text override reasons.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets