Dataset Information

A Tool for Evaluating Medication Alerting Systems: Development and Initial Assessment.

ABSTRACT:

Background

It is well known that recommendations from electronic medication alerts are seldom accepted or acted on by users. Key factors affecting the effectiveness of medication alerts include system usability and alert design. Thus, human factors principles that apply knowledge of human capabilities and limitations are increasingly used in the design of health technology to improve the usability of systems.

Objective

This study aims to evaluate a newly developed evidence-based self-assessment tool that allows the valid and reliable evaluation of computerized medication alerting systems. This tool was developed to be used by hospital staff with detailed knowledge of their hospital's computerized provider order entry system and alerts to identify and address potential system deficiencies. In this initial assessment, we aim to determine whether the items in the tool can measure compliance of medication alerting systems with human factors principles of design, the tool can be consistently used by multiple users to assess the same system, and the items are easy to understand and perceived to be useful for assessing medication alerting systems.

Methods

The Tool for Evaluating Medication Alerting Systems (TEMAS) was developed based on human factors design principles and consisted of 66 items. In total, 18 staff members recruited across 6 hospitals used the TEMAS to assess their medication alerting systems. Data collected from participant assessments were used to evaluate the validity, reliability, and usability of the TEMAS. Validity was assessed by comparing the results of the TEMAS with those of prior in-house evaluations. Reliability was measured using Krippendorff α to determine agreement among assessors. A 7-item survey was used to determine usability.

Results

The participants reported mostly negative (n=8) and neutral (n=7) perceptions of alerts in their medication alerting system. However, the validity of the TEMAS could not be directly tested, as participants were unaware of any results from prior in-house evaluations. The reliability of the TEMAS, as measured by Krippendorff α, was low to moderate (range 0.26-0.46); however, participant feedback suggests that individuals' knowledge of the system varied according to their professional background. In terms of usability, 61% (11/18) of participants reported that the TEMAS items were generally easy to understand; however, participants suggested the revision of 22 items to improve clarity.

Conclusions

This initial assessment of the TEMAS allowed the identification of its components that required modification to improve usability and usefulness. It also revealed that for the TEMAS to be effective in facilitating a comprehensive assessment of a medication alerting system, it should be completed by a multidisciplinary team of hospital staff from both clinical and technical backgrounds to maximize their knowledge of systems.

SUBMITTER: Zheng WY

PROVIDER: S-EPMC8325080 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:Medication non-adherence remains a significant barrier in achieving better health outcomes for patients with chronic diseases. Previous self-reported medication adherence tools were not developed in the context of the Malaysia population. The most commonly used tool, MMAS-8, is no longer economical because it requires a license and currently every form used is charged. Hence, there is a need to develop and validate a new medication adherence tool. The Malaysia Medication Adherence Assessment Tool (MyMAAT) was developed by a multidisciplinary team with expertise in medication adherence and health literacy. The face and content validities of the MyMAAT was established by a panel of experts. A total of 495 patients with type 2 diabetes were recruited from the Ministry of Health facilities consisting of five hospitals and five primary health clinics. A test-retest was conducted on 42 of the patients one week following their first data collection. Exploratory factor analysis was performed to evaluate the validity of the MyMAAT. The final item for MyMAAT was compared with SEAMS, HbA1c%, Medication Possession ratio (MPR) score, and pharmacist's subjective assessment for its hypothesis testing validity. The MyMAAT-12 achieved acceptable internal consistency (Cronbach's alpha = 0.910) and stable reliability as the test-retest score showed good to excellent correlation (Spearman's rho = 0.96, p = 0.001). The MyMAAT has significant moderate association with SEAMS (Spearman's rho = 0.44, p = < 0.001) and significant relationship with HbA1c (< 8% and ≥ 8%) (χ2(1) = 13.4, p < 0.001), MPR (χ2(1) = 13.6, p < 0.001) and pharmacist's subjective assessment categories (χ2(1) = 31, p < 0.001). The sensitivity of MyMAAT-12, tested against HbA1c% was 72.9% while its specificity was 43%. This study demonstrates that the MyMAAT-12 together with other methods of assessment may make a better screening tool to identify patients who were non-adherence to their medications.

Project description:BackgroundFarming is a stressful occupation, and a growing body of research shows that farm stressors are associated with poor mental health. To date, there are few methodologically sound surveys that assess farm stressors, and none have been validated for the Canadian context. Our study aimed to: (a) investigate the types of stressors experienced by farmers, (b) develop a farm stress assessment tool and test its factor structure and internal consistency, and (c) assess its criterion-related validity to self-reported levels of anxiety, depression, burnout, and resilience among farmers.MethodsWe developed a 20-item survey based on a review of the literature, examining existing farm stress surveys, and consulting 10 farmers and agricultural industry experts. Then, a convenience sample of farmers living in Alberta, Canada (Sample 1, N = 354) completed a questionnaire containing the 20-item farm stress survey and four validated measures that assessed depression, anxiety, burnout, and resilience. Sample 1 was used to assess the factor structure using exploratory factor analysis (EFA), internal consistency, and criterion-validity of the survey. Next, a convenience sample of farmers living outside of Alberta (Sample 2, N = 138) was used to evaluate the factor structure of the survey using confirmatory factor analysis (CFA).ResultsThe results of the EFA revealed five underlying dimensions of farm stressors: Unexpected work disruptions, Agricultural hazards, Farm and financial planning, Isolation, and Regulations and public pressure. The subscales accounted for 61.6% of the variance, and the internal consistency (Cronbach's alpha) ranged from 0.66 to.75. Subscale correlations were below 0.44, indicating evidence of discriminant validity. Correlations between the five subscales and the four mental health outcome variables supported the criterion-related validity of the survey. The results of the CFA indicated that the data fit the model, and fit was further improved by correlating one pair of error terms.ConclusionsPreliminary analysis of our Farmer Stress Assessment Tool (FSAT) suggests it is a reliable and valid instrument for measuring a range of stressors farmers face. Implications for policy and community-based mental health interventions that help farmers manage the enduring stressors of agriculture is discussed.

Project description:BackgroundMedication is the most common intervention in health care, and the number of online consumer information systems within the pharmaceutical sector is increasing. However, online consumer information systems can be a barrier for users, imposing information asymmetries between stakeholders.ObjectiveThe objective of this study was to quantify and compare the usability of an online consumer medication information system (OCMIS) against a reference implementation based on an interoperable information model for patients, physicians, and pharmacists.MethodsQuantitative and qualitative data were acquired from patients, physicians, and pharmacists in this online usability study. We administered 3 use cases and a post hoc questionnaire per user. Quantitative usability data including effectiveness (task success), efficiency (task time), and user satisfaction (system usability scale [SUS]) was complemented by qualitative and demographic data. Users evaluated 6 existing systems and 1 reference implementation of an OCMIS.ResultsA total of 137 patients, 81 physicians, and 68 pharmacists participated in this study. Task success varied from 84% to 92% in patients, 66% to 100% in physicians, and 50% to 91% in pharmacists. Task completion time decreased over the course of the study for all but 2 OCMIS within the patient group. Due to an assumed nonnormal distribution of SUS scores, within-group comparison was done using the Kruskal-Wallis test. Patients showed differences in SUS scores (P=.02) and task time (P=.03), while physicians did not have significant differences in SUS scores (P=.83) and task time (P=.72). For pharmacists, a significant difference in SUS scores (P<.001) and task time (P=.007) was detected.ConclusionsThe vendor-neutral reference implementation based on an interoperable information model was proven to be a promising approach that was not inferior to existing solutions for patients and physicians. For pharmacists, it exceeded user satisfaction scores compared to other OCMIS. This data-driven approach based on an interoperable information model enables the development of more user-tailored features to increase usability. This fosters data democratization and empowers stakeholders within the pharmaceutical sector.

Project description:IntroductionTeaching medication history taking or medication reconciliation to students requires practice for students to achieve competency. Practice makes students more confident with the process, but multiple practice opportunities require multiple cases, and creating these new cases can be a tedious and time-consuming process.MethodsThe Medication Mysteries Infinite Case Tool was designed to produce random patient cases using game-like features to allow students to practice medication history taking and medication reconciliation without the need to use and train standardized patients. The tool was created using a random draw card-based system to determine patient personality attributes, drugs they are taking, and confusions they have about their drug-taking behavior. This tool is used in a lab dedicated for the purpose of practicing medication history taking with students being assessed via simulation with standardized patients. This tool is currently used at the University of Florida College of Pharmacy as part of a training program for first-year pharmacy students.ResultsSince 2011, seven classes of first- and second-year pharmacy students have participated in this lab. Each year's class contained an average of 280 students divided into lab groups of 18-24 students. In our initial offering of the lab and assessment, 200 students on three campuses completed the individual assessments following the laboratory session. Fifty-eight percent achieved excellence, and 39% achieved competence on the individual assessment. Only 3% were assessed as being deficient on their performance and were required to repeat the assessment. Overall, 86.8% agreed or strongly agreed that the MMICT was an excellent way to experience how to reconcile medication.DiscussionStudents enjoy the practice and become proficient with the skills they learn through this process as evidenced by increased self-efficacy and achieved competence on a standardized assessment. The tool and the research associated with the outcome were awarded with the 2012 American Association of Colleges of Pharmacy Innovations in Teaching Award.

Project description:OBJECTIVE:The aim of the study was to develop a multifactorial tool for assessment of sedation in dogs. METHODS:Following a modified Delphi method, thirty-eight veterinary anaesthetists were contacted to describe the following levels of awareness: no-sedation, light, moderate, profound sedation and excitation. The answers were summarized in descriptors for each level. A questionnaire was created with all the variables obtained from the descriptors. The questionnaire was returned to the panel of anaesthetists to be used before and after real sedations in conjunction with the previous 5-point categorical scale. Data obtained were analysed using the classification-tree and random-forest methods. RESULTS:Twenty-three anaesthetists (60%) replied with descriptions. The descriptors and study variables were grouped in categories: state-of-mind, posture, movements, stimuli-response, behaviour, response-to-restraint, muscle tone, physiological data, facial-expression, eye position, eyelids, pupils, vocalization and feasibility-to-perform-intended-procedure. The anaesthetists returned 205 completed questionnaires. The levels of awareness reported by the anaesthetists were: no sedation in 92, mild (26), moderate (37) and profound in 50 cases. The classification-tree detected 6 main classifying variables: change in posture, response-to-restraint, head-elevation, response-to-toe-pinching, response-to-name, and movements. The random-forest found that the following variables: change in posture, response-to-restraint, head-elevation, response-to-name, movements, posture, response-to-toe-pinching, demeanour, righting-reflex and response-to-handclap, were classified correctly in 100% awake, 62% mild, 70% moderate and 86% of profound sedation cases. DISCUSSION AND CONCLUSION:The questionnaire and methods developed here classified correctly the level of sedation in most cases. Further studies are needed to evaluate the validity of this tool in the clinical and research setting.