Dataset Information

Cranky comments: detecting clinical decision support malfunctions through free-text override reasons.

ABSTRACT: Background:Rule-base clinical decision support alerts are known to malfunction, but tools for discovering malfunctions are limited. Objective:Investigate whether user override comments can be used to discover malfunctions. Methods:We manually classified all rules in our database with at least 10 override comments into 3 categories based on a sample of override comments: "broken," "not broken, but could be improved," and "not broken." We used 3 methods (frequency of comments, cranky word list heuristic, and a Naïve Bayes classifier trained on a sample of comments) to automatically rank rules based on features of their override comments. We evaluated each ranking using the manual classification as truth. Results:Of the rules investigated, 62 were broken, 13 could be improved, and the remaining 45 were not broken. Frequency of comments performed worse than a random ranking, with precision at 20 of 8 and AUC?=?0.487. The cranky comments heuristic performed better with precision at 20 of 16 and AUC?=?0.723. The Naïve Bayes classifier had precision at 20 of 17 and AUC?=?0.738. Discussion:Override comments uncovered malfunctions in 26% of all rules active in our system. This is a lower bound on total malfunctions and much higher than expected. Even for low-resource organizations, reviewing comments identified by the cranky word list heuristic may be an effective and feasible way of finding broken alerts. Conclusion:Override comments are a rich data source for finding alerts that are broken or could be improved. If possible, we recommend monitoring all override comments on a regular basis.

SUBMITTER: Aaron S

PROVIDER: S-EPMC6308015 | biostudies-literature | 2019 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Cranky comments: detecting clinical decision support malfunctions through free-text override reasons.

Aaron Skye S McEvoy Dustin S DS Ray Soumi S Hickman Thu-Trang T TT Wright Adam A

Journal of the American Medical Informatics Association : JAMIA 20190101 1

<h4>Background</h4>Rule-base clinical decision support alerts are known to malfunction, but tools for discovering malfunctions are limited.<h4>Objective</h4>Investigate whether user override comments can be used to discover malfunctions.<h4>Methods</h4>We manually classified all rules in our database with at least 10 override comments into 3 categories based on a sample of override comments: "broken," "not broken, but could be improved," and "not broken." We used 3 methods (frequency of comments ...[more]

PMID: 30590557

Similar Datasets

Project description:OBJECTIVES:To analyse free-text responses from the first Scottish Cancer Patient Experience Survey (SCPES) to understand patients' experiences of care, identify valued aspects and areas for improvement. DESIGN:Inductive thematic analysis of seven free-text comment boxes covering all stages of the cancer experience, from a national cohort survey. SETTING AND PARTICIPANTS:Adult cancer patients diagnosed across all Health Boards in Scotland between July 2013 and March 2014, and who had an inpatient stay or hospital visit between January and September 2014. 2663 respondents (of n=4835?survey respondents) provided 6961 free-text comments. MAIN OUTCOME MEASURES:Positive and negative themes of patients' experiences. Differences in the proportion of positive to negative comments by demographics. METHODS:Data were analysed as follows: (1) comments were initially categorised at a high level (eg, positive, negative, miscellaneous, etc); (2) inductive codes were derived and applied to all relevant comments; (3) codes sharing similar meaning were amalgamated into subthemes, and code frequencies were measured; (4) subthemes were mapped into overarching themes; (5) difference in the proportion of positive to negative comments by demographics were analysed using ?2 tests. RESULTS:Participants made more positive than negative comments (1:0.78). Analysis highlighted the importance to patients of Feeling that Individual Needs Are Met and Feeling Confident Within the System. Comments also provided insight into how Processes and Structures within the system of care can negatively impact on patients' experience. Particular issues were identified with care experiences in the lead-up to diagnosis. CONCLUSIONS:This analysis provides a detailed understanding of patients' cancer care experiences, therefore indicating what aspects matter in those experiences. Although the majority of comments were positive, there were a significant number of negative comments, especially about the lead-up to diagnosis. Comments suggest patients would value greater integration of care from services involved in their treatment for cancer.

Project description:BackgroundSocial and behavioral determinants of health (SBDH) are environmental and behavioral factors that often impede disease management and result in sexually transmitted infections. Despite their importance, SBDH are inconsistently documented in electronic health records (EHRs) and typically collected only in an unstructured format. Evidence suggests that structured data elements present in EHRs can contribute further to identify SBDH in the patient record.ObjectiveExplore the automated inference of both the presence of SBDH documentation and individual SBDH risk factors in patient records. Compare the relative ability of clinical notes and structured EHR data, such as laboratory measurements and diagnoses, to support inference.MethodsWe attempt to infer the presence of SBDH documentation in patient records, as well as patient status of 11 SBDH, including alcohol abuse, homelessness, and sexual orientation. We compare classification performance when considering clinical notes only, structured data only, and notes and structured data together. We perform an error analysis across several SBDH risk factors.ResultsClassification models inferring the presence of SBDH documentation achieved good performance (F1 score: 92.7-78.7; F1 considered as the primary evaluation metric). Performance was variable for models inferring patient SBDH risk status; results ranged from F1?=?82.7 for LGBT (lesbian, gay, bisexual, and transgender) status to F1?=?28.5 for intravenous drug use. Error analysis demonstrated that lexical diversity and documentation of historical SBDH status challenge inference of patient SBDH status. Three of five classifiers inferring topic-specific SBDH documentation and 10 of 11 patient SBDH status classifiers achieved highest performance when trained using both clinical notes and structured data.ConclusionOur findings suggest that combining clinical free-text notes and structured data provide the best approach in classifying patient SBDH status. Inferring patient SBDH status is most challenging among SBDH with low prevalence and high lexical diversity.

Dataset Information

Cranky comments: detecting clinical decision support malfunctions through free-text override reasons.

Publications

Cranky comments: detecting clinical decision support malfunctions through free-text override reasons.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets