Dataset Information

Multi-method laboratory user evaluation of an actionable clinical performance information system: Implications for usability and patient safety.

ABSTRACT:

Introduction

Electronic audit and feedback (e-A&F) systems are used worldwide for care quality improvement. They measure health professionals' performance against clinical guidelines, and some systems suggest improvement actions. However, little is known about optimal interface designs for e-A&F, in particular how to present suggested actions for improvement. We developed a novel theory-informed system for primary care (the Performance Improvement plaN GeneratoR; PINGR) that covers the four principal interface components: clinical performance summaries; patient lists; detailed patient-level information; and suggested actions. As far as we are aware, this is the first report of an e-A&F system with all four interface components.

Objectives

(1) Use a combination of quantitative and qualitative methods to evaluate the usability of PINGR with target end-users; (2) refine existing design recommendations for e-A&F systems; (3) determine the implications of these recommendations for patient safety.

Methods

We recruited seven primary care physicians to perform seven tasks with PINGR, during which we measured on-screen behaviour and eye movements. Participants subsequently completed usability questionnaires, and were interviewed in-depth. Data were integrated to: gain a more complete understanding of usability issues; enhance and explain each other's findings; and triangulate results to increase validity.

Results

Participants committed a median of 10 errors (range 8-21) when using PINGR's interface, and completed a median of five out of seven tasks (range 4-7). Errors violated six usability heuristics: clear response options; perceptual grouping and data relationships; representational formats; unambiguous description; visually distinct screens for confusable items; and workflow integration. Eye movement analysis revealed the integration of components largely supported effective user workflow, although the modular design of clinical performance summaries unnecessarily increased cognitive load. Interviews and questionnaires revealed PINGR is user-friendly, and that improved information prioritisation could further promote useful user action.

Conclusions

Comparing our results with the wider usability literature we refine a previously published set of interface design recommendations for e-A&F. The implications for patient safety are significant regarding: user engagement; actionability; and information prioritisation. Our results also support adopting multi-method approaches in usability studies to maximise issue discovery and the credibility of findings.

SUBMITTER: Brown B

PROVIDER: S-EPMC5766660 | biostudies-literature | 2018 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Multi-method laboratory user evaluation of an actionable clinical performance information system: Implications for usability and patient safety.

Brown Benjamin B Balatsoukas Panos P Williams Richard R Sperrin Matthew M Buchan Iain I

Journal of biomedical informatics 20171113

<h4>Introduction</h4>Electronic audit and feedback (e-A&F) systems are used worldwide for care quality improvement. They measure health professionals' performance against clinical guidelines, and some systems suggest improvement actions. However, little is known about optimal interface designs for e-A&F, in particular how to present suggested actions for improvement. We developed a novel theory-informed system for primary care (the Performance Improvement plaN GeneratoR; PINGR) that covers the f ...[more]

PMID: 29146562

Similar Datasets

Project description:BackgroundNumerous studies have revealed widespread clinician frustration with the usability of electronic health records (EHRs) that is counterproductive to adoption of EHR systems to meet the aims of health-care reform. With poor system usability comes increased risk of negative unintended consequences. Usability issues could lead to user error and workarounds that have the potential to compromise patient safety and negatively impact the quality of care.[1] While there is ample research on EHR usability, there is little information on the usability of laboratory information systems (LISs). Yet, LISs facilitate the timely provision of a great deal of the information needed by physicians to make patient care decisions.[2] Medical and technical advances in genomics that require processing of an increased volume of complex laboratory data further underscore the importance of developing user-friendly LISs. This study aims to add to the body of knowledge on LIS usability.MethodsA survey was distributed among LIS users at hospitals across the United States. The survey consisted of the ten-item System Usability Scale (SUS). In addition, participants were asked to rate the ease of performing 24 common tasks with a LIS. Finally, respondents provided comments on what they liked and disliked about using the LIS to provide diagnostic insight into LIS perceived usability.ResultsThe overall mean SUS score of 59.7 for the LIS evaluated is significantly lower than the benchmark of 68 (P < 0.001). All LISs evaluated received mean SUS scores below 68 except for Orchard Harvest (78.7). While the years of experience using the LIS was found to be a statistically significant influence on mean SUS scores, the combined effect of years of experience and LIS used did not account for the statistically significant difference in the mean SUS score between Orchard Harvest and each of the other LISs evaluated.ConclusionsThe results of this study indicate that overall usability of LISs is poor. Usability lags that of systems evaluated across 446 usability surveys.

Project description:BackgroundVirtual reality (VR) is increasingly used as simulation technology in emergency medicine education and training, in particular for training nontechnical skills. Experimental studies comparing teaching and learning in VR with traditional training media often demonstrate the equivalence or even superiority regarding particular variables of learning or training effectiveness.ObjectiveIn the EPICSAVE (Enhanced Paramedic Vocational Training with Serious Games and Virtual Environments) project, a highly immersive room-scaled multi-user 3-dimensional VR simulation environment was developed. In this feasibility study, we wanted to gain initial insights into the training effectiveness and media use factors influencing learning and training in VR.MethodsThe virtual emergency scenario was anaphylaxis grade III with shock, swelling of the upper and lower respiratory tract, as well as skin symptoms in a 5-year-old girl (virtual patient) visiting an indoor family amusement park with her grandfather (virtual agent). A cross-sectional, one-group pretest and posttest design was used to evaluate the training effectiveness and quality of the training execution. The sample included 18 active emergency physicians.ResultsThe 18 participants rated the VR simulation training positive in terms of training effectiveness and quality of the training execution. A strong, significant correlation (r=.53, P=.01) between experiencing presence and assessing training effectiveness was observed. Perceived limitations in usability and a relatively high extraneous cognitive load reduced this positive effect.ConclusionsThe training within the virtual simulation environment was rated as an effective educational approach. Specific media use factors appear to modulate training effectiveness (ie, improvement through "experience of presence" or reduction through perceived limitations in usability). These factors should be specific targets in the further development of this VR simulation training.

Project description:BackgroundThe design complexity of critical care ventilators (CCVs) can lead to use errors and patient harm. In this study, we present the results of a comparison of four CCVs from market leaders, using a rigorous methodology for the evaluation of use safety and user experience of medical devices.MethodsWe carried out a comparative usability study of four CCVs: Hamilton G5, Puritan Bennett 980, Maquet SERVO-U, and Dräger Evita V500. Forty-eight critical care respiratory therapists participated in this fully counterbalanced, repeated measures study. Participants completed seven clinical scenarios composed of 16 tasks on each ventilator. Use safety was measured by percentage of tasks with use errors or close calls (UE/CCs). User experience was measured by system usability and workload metrics, using the Post-Study System Usability Questionnaire (PSSUQ) and the National Aeronautics and Space Administration Task Load Index (NASA-TLX).ResultsNine of 18 post hoc contrasts between pairs of ventilators were significant after Bonferroni correction, with effect sizes between 0.4 and 1.09 (Cohen's d). There were significantly fewer UE/CCs with SERVO-U when compared to G5 (p?=?0.044) and V500 (p?=?0.020). Participants reported higher system usability for G5 when compared to PB980 (p?=?0.035) and higher system usability for SERVO-U when compared to G5 (p?<?0.001), PB980 (p?<?0.001), and V500 (p?<?0.001). Participants reported lower workload for G5 when compared to PB980 (p?<?0.001) and lower workload for SERVO-U when compared to PB980 (p?<?0.001) and V500 (p?<?0.001). G5 scored better on two of nine possible comparisons; SERVO-U scored better on seven of nine possible comparisons. Aspects influencing participants' performance and perception include the low sensitivity of G5's touchscreen and the positive effect from the quality of SERVO-U's user interface design.ConclusionsThis study provides empirical evidence of how four ventilators from market leaders compare and highlights the importance of medical technology design. Within the boundaries of this study, we can infer that SERVO-U demonstrated the highest levels of use safety and user experience, followed by G5. Based on qualitative data, differences in outcomes could be explained by interaction design, quality of hardware components used in manufacturing, and influence of consumer product technology on users' expectations.

Project description:ImportanceDespite the broad adoption and optimization of electronic health record (EHR) systems across the continuum of care, serious usability and safety problems persist.ObjectiveTo assess whether EHR safety performance is associated with EHR frontline user experience in a national sample of hospitals.Design, setting, and participantsThis cross-sectional study included all US adult hospitals that used the National Quality Forum Leapfrog Health IT Safety Measure and also used the ARCH Collaborative EHR User experience survey from January 1, 2017, to January 1, 2019. Data analysis was performed from September 2020 to November 2022.Main outcomes and measuresThe primary outcomes were hospital performance on the Leapfrog Health IT Safety measure (overall and 10 subcomponents) and the ARCH collaborative frontline user experience scores (overall and 8 subcomponents). Ordinary least squares models with survey responses clustered by hospital were used to assess associations between the overall measures and their subcomponents.ResultsThere were 112 hospitals and 5689 frontline user surveys included in the study. Hospitals scored a mean of 0.673 (range, 0.297-0.973) on the Leapfrog Health IT safety measure; the mean ARCH EHR user experience score was 3.377 (range, 1 [best] to 5 [worst]). The adjusted β coefficient between the overall safety score and overall user experience score was 0.011 (95% CI, 0.006-0.016). The ARCH overall score was also significantly associated with 10 subcategory scores of the Leapfrog Health IT safety score, and the overall Leapfrog score was associated with the 8 subcategory scores of the ARCH user experience score.Conclusions and relevanceThis cross-sectional study found a positive association between frontline user-rated EHR usability and EHR safety performance. This finding suggests that improving EHR usability, which is a current well-known pain point for EHR users, could have direct benefits in terms of improved EHR safety.

Project description:BackgroundComputerized clinical decision support systems (CDSSs) are increasingly adopted in health care to optimize resources and streamline patient flow. However, they often lack scientific validation against standard medical care.ObjectiveThe purpose of this study was to assess the performance, safety, and usability of a CDSS in a university hospital emergency department setting in Kuopio, Finland.MethodsPatients entering the emergency department were asked to voluntarily participate in this study. Patients aged 17 years or younger, patients with cognitive impairments, and patients who entered the unit in an ambulance or with the need for immediate care were excluded. Patients completed the CDSS web-based form and usability questionnaire when waiting for the triage nurse's evaluation. The CDSS data were anonymized and did not affect the patients' usual evaluation or treatment. Retrospectively, 2 medical doctors evaluated the urgency of each patient's condition by using the triage nurse's information, and urgent and nonurgent groups were created. The International Statistical Classification of Diseases, Tenth Revision diagnoses were collected from the electronic health records. Usability was assessed by using a positive version of the System Usability Scale questionnaire.ResultsIn total, our analyses included 248 patients. Regarding urgency, the mean sensitivities were 85% and 19%, respectively, for urgent and nonurgent cases when assessing the performance of CDSS evaluations in comparison to that of physicians. The mean sensitivities were 85% and 35%, respectively, when comparing the evaluations between the two physicians. Our CDSS did not miss any cases that were evaluated to be emergencies by physicians; thus, all emergency cases evaluated by physicians were evaluated as either urgent cases or emergency cases by the CDSS. In differential diagnosis, the CDSS had an exact match accuracy of 45.5% (97/213). The usability was good, with a mean System Usability Scale score of 78.2 (SD 16.8).ConclusionsIn a university hospital emergency department setting with a large real-world population, our CDSS was found to be equally as sensitive in urgent patient cases as physicians and was found to have an acceptable differential diagnosis accuracy, with good usability. These results suggest that this CDSS can be safely assessed further in a real-world setting. A CDSS could accelerate triage by providing patient-provided data in advance of patients' initial consultations and categorize patient cases as urgent and nonurgent cases upon patients' arrival to the emergency department.

Project description:BackgroundDigital health interventions such as tailored websites are emerging as valuable tools to provide individualized exercise and behavioral change information for individuals diagnosed with cancer.ObjectiveThe aim of this study is to investigate and iteratively refine the acceptability and usability of a web-based exercise intervention (ExerciseGuide) for men with metastatic prostate cancer and determine how well individuals can replicate the video-based exercise prescription.MethodsA laboratory-based multi-methods design was used, incorporating questionnaires, think-aloud tests, interviews, and movement screening among 11 men aged 63 to 82 years with metastatic prostate cancer. Overall, 9 participants were undergoing androgen deprivation therapy, and 2 were completing chemotherapy. Data were collected in two waves, with changes made for quality improvement after participant 5.ResultsThe intervention's usability score was deemed moderate overall but improved after modifications (from 60, SD 2.9 to 69.6, SD 2.2 out of 100). Overall, the participants found the intervention acceptable, with scores improving from wave 1 (24.2, SD 1.1 out of 30) to wave 2 (26.3, SD 2.1 out of 30). The personalized multimodal exercise prescription and computer-tailored education were seen as valuable. After wave 1, website navigation videos were added, medical terminology was simplified, and a telehealth component was included after expert real-time telehealth support was requested. Wave 2 changes included the added variety for aerobic exercise modes, reduced computer-tailoring question loads, and improved consistency of style and grammar. Finally, the participants could replicate the resistance exercise videos to a satisfactory level as judged by the movement screen; however, additional technique cueing within the videos is recommended to address safety concerns.ConclusionsThe acceptability and usability of ExerciseGuide were deemed satisfactory. Various problems were identified and resolved. Notably, the participants requested the inclusion of personalized expert support through telehealth. The resistance training algorithms were shown to provide appropriate content safely, and the users could replicate the exercise technique unaided to a satisfactory level. This study has optimized the ExerciseGuide intervention for further investigation in this population.Trial registrationAustralian New Zealand Clinical Trials Registry (ANZCTR) ACTRN12618001978257; https://anzctr.org.au/Trial/Registration/TrialReview.aspx?ACTRN=12618001978257.

Dataset Information

Multi-method laboratory user evaluation of an actionable clinical performance information system: Implications for usability and patient safety.

Introduction

Objectives

Methods

Results

Conclusions

Publications

Multi-method laboratory user evaluation of an actionable clinical performance information system: Implications for usability and patient safety.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets