Usability Flaws in Medication Alerting Systems: Impact on Usage and Work System.
Ontology highlight
ABSTRACT: Objectives
Previous research has shown that medication alerting systems face usability issues. There has been no previous attempt to systematically explore the consequences of usability flaws in such systems on users (i.e. usage problems) and work systems (i.e. negative outcomes). This paper aims at exploring and synthesizing the consequences of usability flaws in terms of usage problems and negative outcomes on the work system.Methods
A secondary analysis of 26 papers included in a prior systematic review of the usability flaws in medication alerting was performed. Usage problems and negative outcomes were extracted and sorted. Links between usability flaws, usage problems, and negative outcomes were also analyzed.Results
Poor usability generates a large variety of consequences. It impacts the user from a cognitive, behavioral, emotional, and attitudinal perspective. Ultimately, usability flaws have negative consequences on the workflow, the effectiveness of the technology, the medication management process, and, more importantly, patient safety. Only few complete pathways leading from usability flaws to negative outcomes were identified.Conclusion
Usability flaws in medication alerting systems impede users, and ultimately their work system, and negatively impact patient safety. Therefore, the usability dimension may act as a hidden explanatory variable that could explain, at least partly, the (absence of) intended outcomes of new technology.
SUBMITTER: Marcilly R
PROVIDER: S-EPMC4587050 | biostudies-literature | 2015 Aug
REPOSITORIES: biostudies-literature
Publications
Usability Flaws in Medication Alerting Systems: Impact on Usage and Work System.
Marcilly R R Ammenwerth E E Roehrer E E Pelayo S S Vasseur F F Beuscart-Zéphir M-C MC
Yearbook of medical informatics 20150630 1
<h4>Objectives</h4>Previous research has shown that medication alerting systems face usability issues. There has been no previous attempt to systematically explore the consequences of usability flaws in such systems on users (i.e. usage problems) and work systems (i.e. negative outcomes). This paper aims at exploring and synthesizing the consequences of usability flaws in terms of usage problems and negative outcomes on the work system.<h4>Methods</h4>A secondary analysis of 26 papers included i ...[more]
Similar Datasets
Project description:BackgroundUsability flaws in medication alerting systems may have a negative impact on clinical use and patient safety. In order to prevent the release of alerting systems that contain such flaws, it is necessary to provide designers and evaluators with evidence-based usability design principles. The objective of the present study was to develop a comprehensive, structured list of evidence-based usability design principles for medication alerting systems.MethodsNine sets of design principles for medication alerting systems were analyzed, summarized, and structured. We then matched the summarized principles with a list of usability flaws in order to determine the level of underlying evidence.ResultsFifty-eight principles were summarized from the literature and two additional principles were defined, so that each flaw was matched with a principle. We organized the 60 summarized usability design principles into 6 meta-principles, 38 principles, and 16 sub-principles. Only 15 principles were not matched with a usability flaw. The 6 meta-principles respectively covered the improvement of the signal-to-noise ratio, the support for collaborative working, the fit with a clinician's workflow, the data display, the transparency of the alerting system, and the actionable tools to be provided within an alert.ConclusionsIt is possible to develop an evidence-based, structured, comprehensive list of usability design principles that are specific to medication alerting systems and are illustrated by the corresponding usability flaws. This list represents an improvement over the current literature. Each principle is now associated with the best available evidence of its violation. This knowledge may help to improve the usability of medication alerting systems and, ultimately, decrease the harmful consequences of the systems' usability flaws.
| S-EPMC6057098 | biostudies-literature
Project description:BackgroundIt is well known that recommendations from electronic medication alerts are seldom accepted or acted on by users. Key factors affecting the effectiveness of medication alerts include system usability and alert design. Thus, human factors principles that apply knowledge of human capabilities and limitations are increasingly used in the design of health technology to improve the usability of systems.ObjectiveThis study aims to evaluate a newly developed evidence-based self-assessment tool that allows the valid and reliable evaluation of computerized medication alerting systems. This tool was developed to be used by hospital staff with detailed knowledge of their hospital's computerized provider order entry system and alerts to identify and address potential system deficiencies. In this initial assessment, we aim to determine whether the items in the tool can measure compliance of medication alerting systems with human factors principles of design, the tool can be consistently used by multiple users to assess the same system, and the items are easy to understand and perceived to be useful for assessing medication alerting systems.MethodsThe Tool for Evaluating Medication Alerting Systems (TEMAS) was developed based on human factors design principles and consisted of 66 items. In total, 18 staff members recruited across 6 hospitals used the TEMAS to assess their medication alerting systems. Data collected from participant assessments were used to evaluate the validity, reliability, and usability of the TEMAS. Validity was assessed by comparing the results of the TEMAS with those of prior in-house evaluations. Reliability was measured using Krippendorff α to determine agreement among assessors. A 7-item survey was used to determine usability.ResultsThe participants reported mostly negative (n=8) and neutral (n=7) perceptions of alerts in their medication alerting system. However, the validity of the TEMAS could not be directly tested, as participants were unaware of any results from prior in-house evaluations. The reliability of the TEMAS, as measured by Krippendorff α, was low to moderate (range 0.26-0.46); however, participant feedback suggests that individuals' knowledge of the system varied according to their professional background. In terms of usability, 61% (11/18) of participants reported that the TEMAS items were generally easy to understand; however, participants suggested the revision of 22 items to improve clarity.ConclusionsThis initial assessment of the TEMAS allowed the identification of its components that required modification to improve usability and usefulness. It also revealed that for the TEMAS to be effective in facilitating a comprehensive assessment of a medication alerting system, it should be completed by a multidisciplinary team of hospital staff from both clinical and technical backgrounds to maximize their knowledge of systems.
| S-EPMC8325080 | biostudies-literature
Project description:BackgroundMedication is the most common intervention in health care, and the number of online consumer information systems within the pharmaceutical sector is increasing. However, online consumer information systems can be a barrier for users, imposing information asymmetries between stakeholders.ObjectiveThe objective of this study was to quantify and compare the usability of an online consumer medication information system (OCMIS) against a reference implementation based on an interoperable information model for patients, physicians, and pharmacists.MethodsQuantitative and qualitative data were acquired from patients, physicians, and pharmacists in this online usability study. We administered 3 use cases and a post hoc questionnaire per user. Quantitative usability data including effectiveness (task success), efficiency (task time), and user satisfaction (system usability scale [SUS]) was complemented by qualitative and demographic data. Users evaluated 6 existing systems and 1 reference implementation of an OCMIS.ResultsA total of 137 patients, 81 physicians, and 68 pharmacists participated in this study. Task success varied from 84% to 92% in patients, 66% to 100% in physicians, and 50% to 91% in pharmacists. Task completion time decreased over the course of the study for all but 2 OCMIS within the patient group. Due to an assumed nonnormal distribution of SUS scores, within-group comparison was done using the Kruskal-Wallis test. Patients showed differences in SUS scores (P=.02) and task time (P=.03), while physicians did not have significant differences in SUS scores (P=.83) and task time (P=.72). For pharmacists, a significant difference in SUS scores (P<.001) and task time (P=.007) was detected.ConclusionsThe vendor-neutral reference implementation based on an interoperable information model was proven to be a promising approach that was not inferior to existing solutions for patients and physicians. For pharmacists, it exceeded user satisfaction scores compared to other OCMIS. This data-driven approach based on an interoperable information model enables the development of more user-tailored features to increase usability. This fosters data democratization and empowers stakeholders within the pharmaceutical sector.
| S-EPMC7301258 | biostudies-literature
Project description:BackgroundMedication guides are required documents to be distributed to patients in order to convey serious risks associated with certain prescribed medicines. Little is known about the effectiveness of this information to adequately inform patients on safe use.ObjectiveTo examine the readability, suitability, and comprehensibility of medication guides, particularly for those with limited literacy.DesignAssessments of suitability and readability of 185 medication guides, and a sub-study examining change in suitability and readability from 2006 to 2010 among 32 of the medication guides (Study 1); 'open book' comprehension assessment of medication guides (Study 2).SettingTwo general internal medicine clinics in Chicago, IL.PatientsFour hundred and forty-nine adults seeking primary care services, ages 18-85.MeasurementsFor Study 1, the Suitability Assessment of Materials (SAM) and Lexile score for readability. For Study 2, a tailored comprehension assessment of content found in three representative medication guides.ResultsThe 185 analyzed medication guides were on average 1923 words (SD = 1022), with a mean reading level of 10-11th grade. Only one medication guide was deemed suitable in SAM analyses. None provided summaries or reviews, or framed the context first, while very few were rated as having made the purpose evident (8 %), or limited the scope of content (22 %). For Study 2, participants' comprehension of medication guides was poor (M = 52.7 % correct responses, SD = 22.6). In multivariable analysis, low and marginal literacy were independently associated with poorer understanding (β = -14.3, 95 % CI -18.0 - -10.6, p < 0.001; low: β = -23.7, 95 % CI -28.3 - -19.0, p < 0.001).ConclusionCurrent medication guides are of little value to patients, as they are too complex and difficult to understand especially for individuals with limited literacy. Explicit guidance is offered for improving these print materials.
| S-EPMC3509312 | biostudies-literature
Project description:ObjectiveTo quantify and compare the time doctors and nurses spent on direct patient care, medication-related tasks, and interactions before and after electronic medication management system (eMMS) introduction.MethodsControlled pre-post, time and motion study of 129 doctors and nurses for 633.2 h on four wards in a 400-bed hospital in Sydney, Australia. We measured changes in proportions of time on tasks and interactions by period, intervention/control group, and profession.ResultseMMS was associated with no significant change in proportions of time spent on direct care or medication-related tasks relative to control wards. In the post-period control ward, doctors spent 19.7% (2 h/10 h shift) of their time on direct care and 7.4% (44.4 min/10 h shift) on medication tasks, compared to intervention ward doctors (25.7% (2.6 h/shift; p=0.08) and 8.5% (51 min/shift; p=0.40), respectively). Control ward nurses in the post-period spent 22.1% (1.9 h/8.5 h shift) of their time on direct care and 23.7% on medication tasks compared to intervention ward nurses (26.1% (2.2 h/shift; p=0.23) and 22.6% (1.9 h/shift; p=0.28), respectively). We found intervention ward doctors spent less time alone (p=0.0003) and more time with other doctors (p=0.003) and patients (p=0.009). Nurses on the intervention wards spent less time with doctors following eMMS introduction (p=0.0001).ConclusionseMMS introduction did not result in redistribution of time away from direct care or towards medication tasks. Work patterns observed on these intervention wards were associated with previously reported significant reductions in prescribing error rates relative to the control wards.
| S-EPMC3822109 | biostudies-literature
Project description:BackgroundElectronic health record (EHR) system transitions are challenging for healthcare organizations. High-volume, safety-critical tasks like barcode medication administration (BCMA) should be evaluated, yet standards for ensuring safety during transition have not been established.ObjectiveIdentify risks in common and problem-prone medication tasks to inform safe transition between BCMA systems and establish benchmarks for future system changes.DesignStaff nurses completed simulation-based usability testing in the legacy system (R1) and new system pre- (R2) and post-go-live (R3). Tasks included (1) Hold/Administer, (2) IV Fluids, (3) PRN Pain, (4) Insulin, (5) Downtime/PRN, and (6) Messaging. Audiovisual recordings of task performance were systematically analyzed for time, navigation, and errors. The System Usability Scale measured perceived usability and satisfaction. Post-simulation interviews captured nurses' qualitative comments and perceptions of the systems.ParticipantsFifteen staff nurses completed 2-3-h simulation sessions. Eleven completed both R1 and R2, and seven completed all three rounds. Clinical experience ranged from novice (< 1 year) to experienced (> 10 years). Practice settings included adult and pediatric patient populations in ICU, stepdown, and acute care departments.Main measuresTask completion rates/times, safety and non-safety-related use errors (interaction difficulties), and user satisfaction.Key resultsOverall success rates remained relatively stable in all tasks except two: IV Fluids task success increased substantially (R1: 17%, R2: 54%, R3: 100%) and Downtime/PRN task success decreased (R1: 92%, R2: 64%, R3: 22%). Among the seven nurses who completed all rounds, overall safety-related errors decreased 53% from R1 to R3 and 50% from R2 to R3, and average task times for successfully completed tasks decreased 22% from R1 to R3 and 38% from R2 to R3.ConclusionsUsability testing is a reasonable approach to compare different BCMA tasks to anticipate transition problems and establish benchmarks with which to monitor and evaluate system changes going forward.
| S-EPMC10593701 | biostudies-literature
Project description:BackgroundRapid response systems (RRSs) are essential components of patient safety systems; however, limited evidence exists regarding their effectiveness and optimal structures. We aimed to assess the activation patterns and outcomes of RRS implementation with/without a real-time automatic alerting system (AAS) based on electronic medical records (EMRs).MethodsWe retrospectively analyzed clinical data of patients for whom the RRS was activated in the surgical wards of a tertiary university hospital. We compared the code rate, in-hospital mortality, unplanned intensive care unit (ICU) admission, and other clinical outcomes before and after applying RRS and AAS as follows: pre-RRS (January 2013-July 2015), RRS without AAS (August 2015-November 2016), and RRS with AAS (December 2016-December 2017).ResultsIn-hospital mortality per 1000 admissions decreased from 15.1 to 12.9 after RRS implementation (p < 0.001). RRS activation per 1000 admissions increased from 14.4 to 26.3 after AAS implementation. The severity of patients' condition calculated using the modified early warning score increased from 2.5 (± 2.1) in the RRS without AAS to 3.6 (± 2.1) (p < 0.001) in the RRS with AAS. The total and preventable code rates and in-hospital mortality rates were comparable between the RRS implementation periods without/with AAS. ICU duration and mortality results improved in patients with RRS activation and unplanned ICU admission. The data of RRS non-activated group remained unaltered during the study.ConclusionsReal-time AAS based on EMRs might help identify unstable patients. Early detection and intervention with RRS may improve patient outcomes.
| S-EPMC8643026 | biostudies-literature
Project description:Early detection of infectious disease is crucial for reducing transmission and facilitating early intervention. We built a real-time smartwatch-based alerting system for the detection of aberrant physiological and activity signals (e.g. resting heart rate, steps) associated with early infection onset at the individual level. Upon applying this system to a cohort of 3,246 participants, we found that alerts were generated for pre-symptomatic and asymptomatic COVID-19 infections in 78% of cases, and pre-symptomatic signals were observed a median of three days prior to symptom onset. Furthermore, by examining over 100,000 survey annotations, we found that other respiratory infections as well as events not associated with COVID-19 (e.g. stress, alcohol consumption, travel) could trigger alerts, albeit at a lower mean period (1.9 days) than those observed in the COVID-19 cases (4.3 days). Thus this system has potential both for advanced warning of COVID-19 as well as a general system for measuring health via detection of physiological shifts from personal baselines. The system is open-source and scalable to millions of users, offering a personal health monitoring system that can operate in real time on a global scale.
| S-EPMC8240687 | biostudies-literature
Project description:BackgroundNumerous studies have revealed widespread clinician frustration with the usability of electronic health records (EHRs) that is counterproductive to adoption of EHR systems to meet the aims of health-care reform. With poor system usability comes increased risk of negative unintended consequences. Usability issues could lead to user error and workarounds that have the potential to compromise patient safety and negatively impact the quality of care.[1] While there is ample research on EHR usability, there is little information on the usability of laboratory information systems (LISs). Yet, LISs facilitate the timely provision of a great deal of the information needed by physicians to make patient care decisions.[2] Medical and technical advances in genomics that require processing of an increased volume of complex laboratory data further underscore the importance of developing user-friendly LISs. This study aims to add to the body of knowledge on LIS usability.MethodsA survey was distributed among LIS users at hospitals across the United States. The survey consisted of the ten-item System Usability Scale (SUS). In addition, participants were asked to rate the ease of performing 24 common tasks with a LIS. Finally, respondents provided comments on what they liked and disliked about using the LIS to provide diagnostic insight into LIS perceived usability.ResultsThe overall mean SUS score of 59.7 for the LIS evaluated is significantly lower than the benchmark of 68 (P < 0.001). All LISs evaluated received mean SUS scores below 68 except for Orchard Harvest (78.7). While the years of experience using the LIS was found to be a statistically significant influence on mean SUS scores, the combined effect of years of experience and LIS used did not account for the statistically significant difference in the mean SUS score between Orchard Harvest and each of the other LISs evaluated.ConclusionsThe results of this study indicate that overall usability of LISs is poor. Usability lags that of systems evaluated across 446 usability surveys.
| S-EPMC5653961 | biostudies-literature
Project description:Omics data sharing is crucial to the biological research community, and the last decade or two has seen a huge rise in collaborative analysis systems, databases, and knowledge bases for omics and other systems biology data. We assessed the "FAIRness" of NASA's GeneLab Data Systems (GLDS) along with four similar kinds of systems in the research omics data domain, using 14 FAIRness metrics. The range of overall FAIRness scores was 6-12 (out of 14), average 10.1, and standard deviation 2.4. The range of Pass ratings for the metrics was 29-79%, Partial Pass 0-21%, and Fail 7-50%. The systems we evaluated performed the best in the areas of data findability and accessibility, and worst in the area of data interoperability. Reusability of metadata, in particular, was frequently not well supported. We relate our experiences implementing semantic integration of omics data from some of the assessed systems for federated querying and retrieval functions, given their shortcomings in data interoperability. Finally, we propose two new principles that Big Data system developers, in particular, should consider for maximizing data accessibility.
| S-EPMC6371294 | biostudies-other