Dataset Information

Health system guidance appraisal--concept evaluation and usability testing.

ABSTRACT:

Background

Health system guidance (HSG) provides recommendations aimed to address health system challenges. However, there is a paucity of methods to direct, appraise, and report HSG. Earlier research identified 30 candidate criteria (concepts) that can be used to evaluate the quality of HSG and guide development and reporting requirements. The objective of this paper was to describe two studies aimed at evaluating the importance of these 30 criteria, design a draft HSG appraisal tool, and test its usability.

Methods

This study involved a two-step survey process. In step 1, respondents rated the 30 concepts for appropriateness to, relevance to, and priority for health system decisions and HSG. This led to a draft tool. In step 2, respondents reviewed HSG documents, appraised them using the tool, and answered a series of questions. Descriptive analyses were computed.

Results

Fifty participants were invited in step 1, and we had a response rate of 82 %. The mean response rates for each concept within each survey question were universally favorable. There was also an overall agreement about the need for a high-quality tool to systematically direct the development, appraisal, and reporting of HSG. Qualitative feedback and a consensus process by the team led to refinements to some of the concepts and the creation of a beta (draft) version of the HSG tool. In step 2, 35 participants were invited and we had a response rate of 74 %. Exploratory analyses showed that the quality of the HSGs reviewed varied as a function of the HSG item and the specific document assessed. A favorable consensus was reached with participants agreeing that the HSG items were easy to understand and easy to apply. Moreover, the overall agreement was high for the usability of the tool to systematically direct the development (85 %), appraisal (92 %), and reporting (81 %) of HSG. From this process, version 1.0 of the HSG appraisal tool was generated complete with 32 items (and their descriptions) and 4 domains.

Conclusions

The final tool, named the Appraisal of Guidelines for Research and Evaluation for Health Systems (AGREE-HS) (version 1), defines expectations of HSG and facilitates informed decisions among policymakers on health system delivery, financial, and governance arrangements.

SUBMITTER: Ako-Arrey DE

PROVIDER: S-EPMC4700602 | biostudies-literature | 2016 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Health system guidance appraisal--concept evaluation and usability testing.

Ako-Arrey Denis E DE Brouwers Melissa C MC Lavis John N JN Giacomini Mita K MK

Implementation science : IS 20160105

<h4>Background</h4>Health system guidance (HSG) provides recommendations aimed to address health system challenges. However, there is a paucity of methods to direct, appraise, and report HSG. Earlier research identified 30 candidate criteria (concepts) that can be used to evaluate the quality of HSG and guide development and reporting requirements. The objective of this paper was to describe two studies aimed at evaluating the importance of these 30 criteria, design a draft HSG appraisal tool, a ...[more]

PMID: 26727892

Similar Datasets

Project description:Health systems guidance (HSG) are systematically developed statements that assist with decisions about options for addressing health systems challenges, including related changes in health systems arrangements. However, the development, appraisal, and reporting of HSG poses unique conceptual and methodological challenges related to the varied types of evidence that are relevant, the complexity of health systems, and the pre-eminence of contextual factors. To address this gap, we are conducting a program of research that aims to create a tool to support the appraisal of HSG and further enhance HSG development and reporting. The focus of this paper was to conduct a knowledge synthesis of the published and grey literatures to determine quality criteria (concepts) relevant for this process.We applied a critical interpretive synthesis (CIS) approach to knowledge synthesis that enabled an iterative, flexible, and dynamic analysis of diverse bodies of literature in order to generate a candidate list of concepts that will constitute the foundational components of the HSG tool. Using our review questions as compasses, we were able to guide the search strategy to look for papers based on their potential relevance to HSG appraisal, development, and reporting. The search strategy included various electronic databases and sources, subject-specific journals, conference abstracts, research reports, book chapters, unpublished data, dissertations, and policy documents. Screening the papers and data extraction was completed independently and in duplicate, and a narrative approach to data synthesis was executed.We identified 43 papers that met eligibility criteria. No existing review was found on this topic, and no HSG appraisal tool was identified. Over one third of the authors implicitly or explicitly identified the need for a high-quality tool aimed to systematically evaluate HSG and contribute to its development/reporting. We identified 30 concepts that may be relevant to the appraisal of HSG and were able to cluster them into three meaningful domains: process principles, content, and context principles.Our study showed the role that the quality criteria play in the development, appraisal, and reporting of HSG and demonstrated the link and resonance within and between the various concepts in the three domains.

Project description:BackgroundFaecal incontinence (FI) is prevalent in 15-20% of elderly individuals and is frequently monitored in clinical trials and practice. Bowel diaries are the most common way to document FI, but, in clinical practice, are mainly used as paper-based versions. Electronic diaries (eDiaries) offer many potential benefits over paper-based diaries. The aim of this study was to develop and test an eDiary to document FI.MethodsWe migrated a paper FI diary to an eDiary app based on the Computer-based Health Evaluation System (CHES). To assess usability, we conducted functionality and usability tests at two time points in a sample of patients with FI. In the first assessment, the eDiary functionalities were tested, patients completed the System Usability Scale (SUS, range 0-100) and compared the paper diary with the eDiary. We set a threshold for minimum acceptable average usability at 70 points. Patients were then instructed to use the eDiary for 2 days at home and contacted to report on their usage and completed the SUS a second time.ResultsWe recruited a sample of N = 14 patients to use the eDiary. All patients were able to use all functionalities of the eDiary and only a few patients with lower technological literacy or access to devices (n = 3) needed initial assistance. The mean usability rating given at the first time point was high with 88 points (SD 18, 95% CI 78.2-96.8) and most patients (n = 10) reported they would prefer the eDiary over the paper-based version. Nine patients (n = 9) participated in the follow-up assessment and the mean SUS rating at the second time point was 97 points (SD 7, 95% CI 92.8-100).ConclusionThe eDiary showed excellent usability scores for the assessment of FI at both assessments. Generally, patients preferred the eDiary over the paper-based version. We recommend the eDiary for usage with patients who own and use a smartphone and discuss potential solutions for patients with lower technological literacy or access.

Project description:BackgroundWearable devices have the potential to promote a healthy lifestyle because of their real-time data monitoring capabilities. However, device usability is a critical factor that determines whether they will be adopted on a large scale. Usability studies on wearable devices are still scarce.ObjectiveThis study aims to compare the functions and attributes of seven mainstream wearable devices and to evaluate their usability.MethodsThe wearable devices selected were the Apple Watch, Samsung Gear S, Fitbit Surge, Jawbone Up3, Mi Band, Huawei Honor B2, and Misfit Shine. A mixed method of feature comparison and a System Usability Scale (SUS) evaluation based on 388 participants was applied; the higher the SUS score, the better the usability of the product.ResultsFor features, all devices had step counting, an activity timer, and distance recording functions. The Samsung Gear S had a unique sports track recording feature and the Huawei Honor B2 had a unique wireless earphone. The Apple Watch, Samsung Gear S, Jawbone Up3, and Fitbit Surge could measure heart rate. All the devices were able to monitor sleep, except the Apple Watch. For product characteristics, including attributes such as weight, battery life, price, and 22 functions such as step counting, activity time, activity type identification, sleep monitoring, and expandable new features, we found a very weak negative correlation between the SUS scores and price (r=-.10, P=.03) and devices that support expandable new features (r=-.11, P=.02), and a very weak positive correlation between the SUS scores and devices that support the activity type identification function (r=.11, P=.02). The Huawei Honor B2 received the highest score of mean 67.6 (SD 16.1); the lowest Apple Watch score was only 61.4 (SD 14.7). No significant difference was observed among brands. The SUS score had a moderate positive correlation with the user's experience (length of time the device was used) (r=.32, P<.001); participants in the medical and health care industries gave a significantly higher score (mean 61.1, SD 17.9 vs mean 68.7, SD 14.5, P=.03).ConclusionsThe functions of wearable devices tend to be homogeneous and usability is similar across various brands. Overall, Mi Band had the lowest price and the lightest weight. Misfit Shine had the longest battery life and most functions, and participants in the medical and health care industries had the best evaluation of wearable devices. The perceived usability of mainstream wearable devices is unsatisfactory and customer loyalty is not high. A consumer's SUS rating for a wearable device is related to their personal situation instead of the device brand. Device manufacturers should put more effort into developing innovative functions and improving the usability of their products by integrating more cognitive behavior change techniques.

Project description:BackgroundElectronic health record (EHR) system transitions are challenging for healthcare organizations. High-volume, safety-critical tasks like barcode medication administration (BCMA) should be evaluated, yet standards for ensuring safety during transition have not been established.ObjectiveIdentify risks in common and problem-prone medication tasks to inform safe transition between BCMA systems and establish benchmarks for future system changes.DesignStaff nurses completed simulation-based usability testing in the legacy system (R1) and new system pre- (R2) and post-go-live (R3). Tasks included (1) Hold/Administer, (2) IV Fluids, (3) PRN Pain, (4) Insulin, (5) Downtime/PRN, and (6) Messaging. Audiovisual recordings of task performance were systematically analyzed for time, navigation, and errors. The System Usability Scale measured perceived usability and satisfaction. Post-simulation interviews captured nurses' qualitative comments and perceptions of the systems.ParticipantsFifteen staff nurses completed 2-3-h simulation sessions. Eleven completed both R1 and R2, and seven completed all three rounds. Clinical experience ranged from novice (< 1 year) to experienced (> 10 years). Practice settings included adult and pediatric patient populations in ICU, stepdown, and acute care departments.Main measuresTask completion rates/times, safety and non-safety-related use errors (interaction difficulties), and user satisfaction.Key resultsOverall success rates remained relatively stable in all tasks except two: IV Fluids task success increased substantially (R1: 17%, R2: 54%, R3: 100%) and Downtime/PRN task success decreased (R1: 92%, R2: 64%, R3: 22%). Among the seven nurses who completed all rounds, overall safety-related errors decreased 53% from R1 to R3 and 50% from R2 to R3, and average task times for successfully completed tasks decreased 22% from R1 to R3 and 38% from R2 to R3.ConclusionsUsability testing is a reasonable approach to compare different BCMA tasks to anticipate transition problems and establish benchmarks with which to monitor and evaluate system changes going forward.

Dataset Information

Health system guidance appraisal--concept evaluation and usability testing.

Background

Methods

Results

Conclusions

Publications

Health system guidance appraisal--concept evaluation and usability testing.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets