Dataset Information

Validity and usability testing of a health systems guidance appraisal tool, the AGREE-HS.

ABSTRACT: BACKGROUND:Health systems guidance (HSG) provides recommendations to address health systems challenges. No tools exist to inform HSG developers and users about the components of high quality HSG and to differentiate between HSG of varying quality. In response, we developed a tool to assist with the development, reporting and appraisal of HSG - the Appraisal of Guidelines for Research and Evaluation-Health Systems (AGREE-HS). This paper reports on the validity, usability and initial measurement properties of the AGREE-HS. METHODS:To establish face validity (Study 1), stakeholders completed a survey about the AGREE-HS and provided feedback on its content and structure. Revisions to the tool were made in response. To establish usability (Study 2), the revised tool was applied to 85 HSG documents and the appraisers provided feedback about their experiences via an online survey. An initial test of the revised tool's measurement properties, including internal consistency, inter-rater reliability and criterion validity, was conducted. Additional revisions to the tool were made in response. RESULTS:In Study 1, the AGREE-HS Overview, User Manual, quality item content and structure, and overall assessment questions were rated favourably. Participants indicated that the AGREE-HS would be useful, feasible to use, and that they would apply it in their context. In Study 2, participants indicated that the quality items were easy to understand and apply, and the User Manual, usefulness and usability of the tool were rated favourably. Study 2 participants also indicated intentions to use the AGREE-HS. CONCLUSIONS:The AGREE-HS comprises a User Manual, five quality items and two overall assessment questions. It is available at agreetrust.org.

SUBMITTER: Brouwers MC

PROVIDER: S-EPMC6011397 | biostudies-literature | 2018 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Validity and usability testing of a health systems guidance appraisal tool, the AGREE-HS.

Brouwers Melissa C MC Brouwers Melissa C MC Ako-Arrey Denis D Spithoff Karen K Vukmirovic Marija M Florez Ivan D ID Lavis John N JN Cluzeau Francoise F Permanand Govin G Bosch-Capblanch Xavier X Chen Yaolong Y

Health research policy and systems 20180620 1

<h4>Background</h4>Health systems guidance (HSG) provides recommendations to address health systems challenges. No tools exist to inform HSG developers and users about the components of high quality HSG and to differentiate between HSG of varying quality. In response, we developed a tool to assist with the development, reporting and appraisal of HSG - the Appraisal of Guidelines for Research and Evaluation-Health Systems (AGREE-HS). This paper reports on the validity, usability and initial measu ...[more]

PMID: 29925394

Similar Datasets

Project description:BackgroundHealth system guidance (HSG) provides recommendations aimed to address health system challenges. However, there is a paucity of methods to direct, appraise, and report HSG. Earlier research identified 30 candidate criteria (concepts) that can be used to evaluate the quality of HSG and guide development and reporting requirements. The objective of this paper was to describe two studies aimed at evaluating the importance of these 30 criteria, design a draft HSG appraisal tool, and test its usability.MethodsThis study involved a two-step survey process. In step 1, respondents rated the 30 concepts for appropriateness to, relevance to, and priority for health system decisions and HSG. This led to a draft tool. In step 2, respondents reviewed HSG documents, appraised them using the tool, and answered a series of questions. Descriptive analyses were computed.ResultsFifty participants were invited in step 1, and we had a response rate of 82 %. The mean response rates for each concept within each survey question were universally favorable. There was also an overall agreement about the need for a high-quality tool to systematically direct the development, appraisal, and reporting of HSG. Qualitative feedback and a consensus process by the team led to refinements to some of the concepts and the creation of a beta (draft) version of the HSG tool. In step 2, 35 participants were invited and we had a response rate of 74 %. Exploratory analyses showed that the quality of the HSGs reviewed varied as a function of the HSG item and the specific document assessed. A favorable consensus was reached with participants agreeing that the HSG items were easy to understand and easy to apply. Moreover, the overall agreement was high for the usability of the tool to systematically direct the development (85 %), appraisal (92 %), and reporting (81 %) of HSG. From this process, version 1.0 of the HSG appraisal tool was generated complete with 32 items (and their descriptions) and 4 domains.ConclusionsThe final tool, named the Appraisal of Guidelines for Research and Evaluation for Health Systems (AGREE-HS) (version 1), defines expectations of HSG and facilitates informed decisions among policymakers on health system delivery, financial, and governance arrangements.

Project description:BackgroundDuring health emergencies, leading healthcare organisations, such as the World Health Organization (WHO), the European Centre for Disease Control and Prevention (ECDC), and the United States Centers for Disease Control and Prevention (CDC), provide guidance for public health response. Previous studies have evaluated clinical practice guidelines (CPGs) produced in response to epidemics or pandemics, yet few have focused on public health guidelines and recommendations. To address this gap, we assessed health systems guidance (HSG) produced by the WHO, the ECDC, and the CDC for the 2009 H1N1 and COVID-19 pandemics.MethodsWe extracted HSG for the H1N1 and COVID-19 pandemics from the organisations' dedicated repositories and websites. After screening the retrieved documents for eligibility, five assessors evaluated them using the Appraisal of Guidelines Research & Evaluation - Health Systems (AGREE-HS) tool to assess the completeness and transparency of reporting according to the five AGREE-HS domains: "Topic", "Participants", "Methods", "Recommendations", and "Implementability".ResultsFollowing the screening process, we included 108 HSG in the analysis. We observed statistically significant differences between the H1N1 and COVID-19 pandemics, with HSG issued during COVID-19 receiving higher AGREE-HS scores. The HSG produced by the CDC had significantly lower overall scores and single-domain scores compared to the WHO and ECDC. However, all HSG scored relatively low, under the median of 40 total points (range = 10-70), indicating incomplete reporting. The HSG produced by all three organisations received a median score <4 (range = 1-7) for the "Participants", "Methods", and "Implementability" domains.ConclusionsThere is still significant progress to be made in the quality and completeness of reporting in HSG issued during pandemics, especially regarding methodological approaches and the composition of the guidance development team. Due to their significant impact and importance for healthcare systems globally, HSG issued during future healthcare crises should adhere to best reporting practices to increase uptake by stakeholders and ensure public trust in healthcare organisations.

Project description:Health systems guidance (HSG) are systematically developed statements that assist with decisions about options for addressing health systems challenges, including related changes in health systems arrangements. However, the development, appraisal, and reporting of HSG poses unique conceptual and methodological challenges related to the varied types of evidence that are relevant, the complexity of health systems, and the pre-eminence of contextual factors. To address this gap, we are conducting a program of research that aims to create a tool to support the appraisal of HSG and further enhance HSG development and reporting. The focus of this paper was to conduct a knowledge synthesis of the published and grey literatures to determine quality criteria (concepts) relevant for this process.We applied a critical interpretive synthesis (CIS) approach to knowledge synthesis that enabled an iterative, flexible, and dynamic analysis of diverse bodies of literature in order to generate a candidate list of concepts that will constitute the foundational components of the HSG tool. Using our review questions as compasses, we were able to guide the search strategy to look for papers based on their potential relevance to HSG appraisal, development, and reporting. The search strategy included various electronic databases and sources, subject-specific journals, conference abstracts, research reports, book chapters, unpublished data, dissertations, and policy documents. Screening the papers and data extraction was completed independently and in duplicate, and a narrative approach to data synthesis was executed.We identified 43 papers that met eligibility criteria. No existing review was found on this topic, and no HSG appraisal tool was identified. Over one third of the authors implicitly or explicitly identified the need for a high-quality tool aimed to systematically evaluate HSG and contribute to its development/reporting. We identified 30 concepts that may be relevant to the appraisal of HSG and were able to cluster them into three meaningful domains: process principles, content, and context principles.Our study showed the role that the quality criteria play in the development, appraisal, and reporting of HSG and demonstrated the link and resonance within and between the various concepts in the three domains.

Project description:BackgroundCommunity Paramedics (CPs) require access to timely blood analysis in the field to guide treatment and transport decisions. Point of care testing (POCT), as opposed to laboratory analysis, may offer a solution, but limited research exists on CP POCT. The purpose of this study was to compare the validity of two devices (Abbott i-STAT® and Alere epoc®) by CPs in the community.MethodsIn a CP programme responding to 6000 annual patient care events, a split sample validation of POCT against traditional laboratory analysis for seven analytes (sodium, potassium, chloride, creatinine, hemoglobin, hematocrit, and glucose) was conducted on a consecutive sample of patients. The difference of proportion of discrepant results between POCT and laboratory was compared using a two sample proportion test. Usability was analysed by survey of CP experience, a linear mixed effects model of Systems Usability Scale (SUS) adjusted for CP clinical and POCT experience, an expert heuristic evaluation of devices, a review of device-logged errors, and coded observations of POCT use during quality control testing.ResultsOf 1649 episodes of care screened for enrollment, 174 required a blood draw, with 108 episodes (62.1%) enrolled from 73 participants. Participants had a mean age of 58.7 years (SD16.3); 49% were female. In 4 of 646 (0.6%) comparisons, POCT reported a critical value but the laboratory did not; with no statistically significant (p = 0.323) difference between i-STAT® (0.9%;95%CI:0.0,1.9%) compared with epoc® (0.3%;95%CI:0.0,0.9%). There were no instances of the laboratory reporting a critical value when POCT did not. In 88 of 1046 (8.4%) comparisons the a priori defined acceptable difference between POCT and the laboratory was exceeded; occurring more often in epoc® (10.7%;95%CI:8.1,13.3%) compared with i-STAT® (6.1%;95%CI:4.1,8.2%)(p = 0.007). Eighteen of 19 CP surveys were returned, with 11/18 (61.1%) preferring i-STAT® over epoc®. The i-STAT® had a higher mean SUS score (higher usability) compared with epoc® (84.0/100 vs. 59.6/100; p = 0.011). There were no statistically significant differences in device logged errors between i-STAT® and epoc® (p = 0.063).ConclusionsCP programmes can expect clinically valid results from POCT. Device usability assessments should be considered with any local implementation as the two POCT systems have different strengths.

Dataset Information

Validity and usability testing of a health systems guidance appraisal tool, the AGREE-HS.

Publications

Validity and usability testing of a health systems guidance appraisal tool, the AGREE-HS.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets