Dataset Information

A Tool for Evaluating Medication Alerting Systems: Development and Initial Assessment.

ABSTRACT:

Background

It is well known that recommendations from electronic medication alerts are seldom accepted or acted on by users. Key factors affecting the effectiveness of medication alerts include system usability and alert design. Thus, human factors principles that apply knowledge of human capabilities and limitations are increasingly used in the design of health technology to improve the usability of systems.

Objective

This study aims to evaluate a newly developed evidence-based self-assessment tool that allows the valid and reliable evaluation of computerized medication alerting systems. This tool was developed to be used by hospital staff with detailed knowledge of their hospital's computerized provider order entry system and alerts to identify and address potential system deficiencies. In this initial assessment, we aim to determine whether the items in the tool can measure compliance of medication alerting systems with human factors principles of design, the tool can be consistently used by multiple users to assess the same system, and the items are easy to understand and perceived to be useful for assessing medication alerting systems.

Methods

The Tool for Evaluating Medication Alerting Systems (TEMAS) was developed based on human factors design principles and consisted of 66 items. In total, 18 staff members recruited across 6 hospitals used the TEMAS to assess their medication alerting systems. Data collected from participant assessments were used to evaluate the validity, reliability, and usability of the TEMAS. Validity was assessed by comparing the results of the TEMAS with those of prior in-house evaluations. Reliability was measured using Krippendorff α to determine agreement among assessors. A 7-item survey was used to determine usability.

Results

The participants reported mostly negative (n=8) and neutral (n=7) perceptions of alerts in their medication alerting system. However, the validity of the TEMAS could not be directly tested, as participants were unaware of any results from prior in-house evaluations. The reliability of the TEMAS, as measured by Krippendorff α, was low to moderate (range 0.26-0.46); however, participant feedback suggests that individuals' knowledge of the system varied according to their professional background. In terms of usability, 61% (11/18) of participants reported that the TEMAS items were generally easy to understand; however, participants suggested the revision of 22 items to improve clarity.

Conclusions

This initial assessment of the TEMAS allowed the identification of its components that required modification to improve usability and usefulness. It also revealed that for the TEMAS to be effective in facilitating a comprehensive assessment of a medication alerting system, it should be completed by a multidisciplinary team of hospital staff from both clinical and technical backgrounds to maximize their knowledge of systems.

SUBMITTER: Zheng WY

PROVIDER: S-EPMC8325080 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:Medication non-adherence remains a significant barrier in achieving better health outcomes for patients with chronic diseases. Previous self-reported medication adherence tools were not developed in the context of the Malaysia population. The most commonly used tool, MMAS-8, is no longer economical because it requires a license and currently every form used is charged. Hence, there is a need to develop and validate a new medication adherence tool. The Malaysia Medication Adherence Assessment Tool (MyMAAT) was developed by a multidisciplinary team with expertise in medication adherence and health literacy. The face and content validities of the MyMAAT was established by a panel of experts. A total of 495 patients with type 2 diabetes were recruited from the Ministry of Health facilities consisting of five hospitals and five primary health clinics. A test-retest was conducted on 42 of the patients one week following their first data collection. Exploratory factor analysis was performed to evaluate the validity of the MyMAAT. The final item for MyMAAT was compared with SEAMS, HbA1c%, Medication Possession ratio (MPR) score, and pharmacist's subjective assessment for its hypothesis testing validity. The MyMAAT-12 achieved acceptable internal consistency (Cronbach's alpha = 0.910) and stable reliability as the test-retest score showed good to excellent correlation (Spearman's rho = 0.96, p = 0.001). The MyMAAT has significant moderate association with SEAMS (Spearman's rho = 0.44, p = < 0.001) and significant relationship with HbA1c (< 8% and ≥ 8%) (χ2(1) = 13.4, p < 0.001), MPR (χ2(1) = 13.6, p < 0.001) and pharmacist's subjective assessment categories (χ2(1) = 31, p < 0.001). The sensitivity of MyMAAT-12, tested against HbA1c% was 72.9% while its specificity was 43%. This study demonstrates that the MyMAAT-12 together with other methods of assessment may make a better screening tool to identify patients who were non-adherence to their medications.

Project description:IntroductionTeaching medication history taking or medication reconciliation to students requires practice for students to achieve competency. Practice makes students more confident with the process, but multiple practice opportunities require multiple cases, and creating these new cases can be a tedious and time-consuming process.MethodsThe Medication Mysteries Infinite Case Tool was designed to produce random patient cases using game-like features to allow students to practice medication history taking and medication reconciliation without the need to use and train standardized patients. The tool was created using a random draw card-based system to determine patient personality attributes, drugs they are taking, and confusions they have about their drug-taking behavior. This tool is used in a lab dedicated for the purpose of practicing medication history taking with students being assessed via simulation with standardized patients. This tool is currently used at the University of Florida College of Pharmacy as part of a training program for first-year pharmacy students.ResultsSince 2011, seven classes of first- and second-year pharmacy students have participated in this lab. Each year's class contained an average of 280 students divided into lab groups of 18-24 students. In our initial offering of the lab and assessment, 200 students on three campuses completed the individual assessments following the laboratory session. Fifty-eight percent achieved excellence, and 39% achieved competence on the individual assessment. Only 3% were assessed as being deficient on their performance and were required to repeat the assessment. Overall, 86.8% agreed or strongly agreed that the MMICT was an excellent way to experience how to reconcile medication.DiscussionStudents enjoy the practice and become proficient with the skills they learn through this process as evidenced by increased self-efficacy and achieved competence on a standardized assessment. The tool and the research associated with the outcome were awarded with the 2012 American Association of Colleges of Pharmacy Innovations in Teaching Award.

Project description:OBJECTIVE:The aim of the study was to develop a multifactorial tool for assessment of sedation in dogs. METHODS:Following a modified Delphi method, thirty-eight veterinary anaesthetists were contacted to describe the following levels of awareness: no-sedation, light, moderate, profound sedation and excitation. The answers were summarized in descriptors for each level. A questionnaire was created with all the variables obtained from the descriptors. The questionnaire was returned to the panel of anaesthetists to be used before and after real sedations in conjunction with the previous 5-point categorical scale. Data obtained were analysed using the classification-tree and random-forest methods. RESULTS:Twenty-three anaesthetists (60%) replied with descriptions. The descriptors and study variables were grouped in categories: state-of-mind, posture, movements, stimuli-response, behaviour, response-to-restraint, muscle tone, physiological data, facial-expression, eye position, eyelids, pupils, vocalization and feasibility-to-perform-intended-procedure. The anaesthetists returned 205 completed questionnaires. The levels of awareness reported by the anaesthetists were: no sedation in 92, mild (26), moderate (37) and profound in 50 cases. The classification-tree detected 6 main classifying variables: change in posture, response-to-restraint, head-elevation, response-to-toe-pinching, response-to-name, and movements. The random-forest found that the following variables: change in posture, response-to-restraint, head-elevation, response-to-name, movements, posture, response-to-toe-pinching, demeanour, righting-reflex and response-to-handclap, were classified correctly in 100% awake, 62% mild, 70% moderate and 86% of profound sedation cases. DISCUSSION AND CONCLUSION:The questionnaire and methods developed here classified correctly the level of sedation in most cases. Further studies are needed to evaluate the validity of this tool in the clinical and research setting.

Project description:BackgroundGroups or institutions funding or conducting systematic reviews in health policy and systems research (HPSR) should prioritise topics according to the needs of policymakers and stakeholders. The aim of this study was to develop and validate a tool to prioritise questions for systematic reviews in HPSR.MethodsWe developed the tool following a four-step approach consisting of (1) the definition of the purpose and scope of tool, (2) item generation and reduction, (3) testing for content and face validity, (4) and pilot testing of the tool. The research team involved international experts in HPSR, systematic review methodology and tool development, led by the Center for Systematic Reviews on Health Policy and Systems Research (SPARK). We followed an inclusive approach in determining the final selection of items to allow customisation to the user's needs.ResultsThe purpose of the SPARK tool was to prioritise questions in HPSR in order to address them in systematic reviews. In the item generation and reduction phase, an extensive literature search yielded 40 relevant articles, which were reviewed by the research team to create a preliminary list of 19 candidate items for inclusion in the tool. As part of testing for content and face validity, input from international experts led to the refining, changing, merging and addition of new items, and to organisation of the tool into two modules. Following pilot testing, we finalised the tool, with 22 items organised in two modules - the first module including 13 items to be rated by policymakers and stakeholders, and the second including 9 items to be rated by systematic review teams. Users can customise the tool to their needs, by omitting items that may not be applicable to their settings. We also developed a user manual that provides guidance on how to use the SPARK tool, along with signaling questions.ConclusionWe have developed and conducted initial validation of the SPARK tool to prioritise questions for systematic reviews in HPSR, along with a user manual. By aligning systematic review production to policy priorities, the tool will help support evidence-informed policymaking and reduce research waste. We invite others to contribute with additional real-life implementation of the tool.

Project description:Background:Measuring the performance of a health system is an essential requirement in creating systems that generate efficient, equitable, patient-focused, accessible and sustainable results. A fundamental requirement for a performance measurement system is the development of an assessment framework within which specific performance measures could be defined and applied regularly. This paper examines the comprehensiveness of Ghana's health system assessment framework called the Holistic Assessment Tool in relation to some of the internationally recognized frameworks. The paper also analyzes trends in the performance of the health system to understand whether or not an improvement has been recorded following the adoption and implementation of the Holistic Assessment Tool. Methods:Mainly secondary data were used in this analysis. Searches were conducted on Google Scholar, PubMed, Scopus and Science Direct between May and July, 2019 for published documents on health system performance assessment. We also obtained unpublished documents from Ghana's Ministry of Health, Ghana Health Service website, and Ghana Statistical Service database. Descriptive statistics were used to examine trends in the performance of the Ghanaian health system. Results:While the tool provides a national framework for evaluating the performance of the Ghana Health system in several domains, the Holistic Assessment Tool does not cover key health system domains such as information systems for health, access to essential medicines, and patient-centeredness. Also, the scope of the assessment program seems limited to the evaluation of the Ministry of Health's annual plans, programs and projects. However, the health system has recorded improvements in population health indicators, such as life expectancy at birth, infant mortality, under-5 mortality, HIV prevalence and disease burden (in terms of disability adjusted life years). Conclusions:The Holistic Assessment Tool is a useful framework, but needs further refinement, both in scope and in conceptual robustness. Future studies should consider exploring factors influencing performance of the Ghanaian health system. Such information will help in strategizing for better and more improvements.