Dataset Information

Efficient clinical evaluation of guideline quality: development and testing of a new tool.

ABSTRACT:

Background

Evaluating the methodological quality of clinical practice guidelines is essential before deciding which ones which could best inform policy or practice. One current method of evaluating clinical guideline quality is the research-focused AGREE II instrument. This uses 23 questions scored 1-7, arranged in six domains, which requires at least two independent testers, and uses a formulaic weighted domain scoring system. Following feedback from time-poor clinicians, policy-makers and managers that this instrument did not suit clinical need, we developed and tested a simpler, shorter, binary scored instrument (the iCAHE Guideline Quality Checklist) designed for single users.

Methods

Content and construct validity, inter-tester reliability and clinical utility were tested by comparing the new iCAHE Guideline Quality Checklist with the AGREE II instrument. Firstly the questions and domains in both instruments were compared. Six randomly-selected guidelines on a similar theme were then assessed by three independent testers with different experience in guideline quality assessment, using both instruments. Per guideline, weighted domain and total AGREE II scores were calculated, using the scoring rubric for three testers. Total iCAHE scores were calculated per guideline, per tester. The linear relationship between iCAHE and AGREE II scores was assessed using Pearson r correlation coefficients. Score differences between testers were assessed for the iCAHE Guideline Quality Checklist.

Results

There were congruent questions in each instrument in four domains (Scope & Purpose, Stakeholder involvement, Underlying evidence/Rigour, Clarity). The iCAHE and AGREE II scores were moderate to strongly correlated for the six guidelines. There was generally good agreement between testers for iCAHE scores, irrespective of their experience. The iCAHE instrument was preferred by all testers, and took significantly less time to administer than the AGREE II instrument. However, the use of only three testers and six guidelines compromised study power, rendering this research as pilot investigations of the psychometric properties of the iCAHE instrument.

Conclusion

The iCAHE Guideline Quality Checklist has promising psychometric properties and clinical utility.

SUBMITTER: Grimmer K

PROVIDER: S-EPMC4033487 | biostudies-literature | 2014 May

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Efficient clinical evaluation of guideline quality: development and testing of a new tool.

Grimmer Karen K Dizon Janine Margarita JM Milanese Steve S King Ellena E Beaton Kate K Thorpe Olivia O Lizarondo Lucylynn L Luker Julie J Machotka Zuzana Z Kumar Saravana S

BMC medical research methodology 20140510

<h4>Background</h4>Evaluating the methodological quality of clinical practice guidelines is essential before deciding which ones which could best inform policy or practice. One current method of evaluating clinical guideline quality is the research-focused AGREE II instrument. This uses 23 questions scored 1-7, arranged in six domains, which requires at least two independent testers, and uses a formulaic weighted domain scoring system. Following feedback from time-poor clinicians, policy-makers ...[more]

PMID: 24885893

Similar Datasets

Project description:BackgroundThe extensive availability and increasing use of mobile apps for nutrition-based health interventions makes evaluation of the quality of these apps crucial for integration of apps into nutritional counseling.ObjectiveThe goal of this research was the development, validation, and reliability testing of the app quality evaluation (AQEL) tool, an instrument for evaluating apps' educational quality and technical functionality.MethodsItems for evaluating app quality were adapted from website evaluations, with additional items added to evaluate the specific characteristics of apps, resulting in 79 initial items. Expert panels of nutrition and technology professionals and app users reviewed items for face and content validation. After recommended revisions, nutrition experts completed a second AQEL review to ensure clarity. On the basis of 150 sets of responses using the revised AQEL, principal component analysis was completed, reducing AQEL into 5 factors that underwent reliability testing, including internal consistency, split-half reliability, test-retest reliability, and interrater reliability (IRR). Two additional modifiable constructs for evaluating apps based on the age and needs of the target audience as selected by the evaluator were also tested for construct reliability. IRR testing using intraclass correlations (ICC) with all 7 constructs was conducted, with 15 dietitians evaluating one app.ResultsDevelopment and validation resulted in the 51-item AQEL. These were reduced to 25 items in 5 factors after principal component analysis, plus 9 modifiable items in two constructs that were not included in principal component analysis. Internal consistency and split-half reliability of the following constructs derived from principal components analysis was good (Cronbach alpha >.80, Spearman-Brown coefficient >.80): behavior change potential, support of knowledge acquisition, app function, and skill development. App purpose split half-reliability was .65. Test-retest reliability showed no significant change over time (P>.05) for all but skill development (P=.001). Construct reliability was good for items assessing age appropriateness of apps for children, teens, and a general audience. In addition, construct reliability was acceptable for assessing app appropriateness for various target audiences (Cronbach alpha >.70). For the 5 main factors, ICC (1,k) was >.80, with a P value of <.05. When 15 nutrition professionals evaluated one app, ICC (2,15) was .98, with a P value of <.001 for all 7 constructs when the modifiable items were specified for adults seeking weight loss support.ConclusionsOur preliminary effort shows that AQEL is a valid, reliable instrument for evaluating nutrition apps' qualities for clinical interventions by nutrition clinicians, educators, and researchers. Further efforts in validating AQEL in various contexts are needed.

Project description:Importance:Clinical practice guidelines (CPGs) may lack rigor and suitability to the setting in which they are to be applied. Methods to yield clinical practice guideline recommendations that are credible and implementable remain to be determined. Objective:To describe the development of AGREE-REX (Appraisal of Guidelines Research and Evaluation-Recommendations Excellence), a tool designed to evaluate the quality of clinical practice guideline recommendations. Design, Setting, and Participants:A cross-sectional study of 322 international stakeholders representing CPG developers, users, and researchers was conducted between December 2015 and March 2019. Advertisements to participate were distributed through professional organizations as well as through the AGREE Enterprise social media accounts and their registered users. Exposures:Between 2015 and 2017, participants appraised 1 of 161 CPGs using the Draft AGREE-REX tool and completed the AGREE-REX Usability Survey. Main Outcomes and Measures:Usability and measurement properties of the tool were assessed with 7-point scales (1 indicating strong disagreement and 7 indicating strong agreement). Internal consistency of items was assessed with the Cronbach ?, and the Spearman-Brown reliability adjustment was used to calculate reliability for 2 to 5 raters. Results:A total of 322 participants (202 female participants [62.7%]; 83 aged 40-49 years [25.8%]) rated the survey items (on a 7-point scale). All 11 items were rated as easy to understand (with a mean [SD] ranging from 5.2 [1.38] for the alignment of values item to 6.3 [0.87] for the evidence item) and easy to apply (with a mean [SD] ranging from 4.8 [1.49] for the alignment of values item to 6.1 [1.07] for the evidence item). Participants provided favorable feedback on the tool's instructions, which were considered clear (mean [SD], 5.8 [1.06]), helpful (mean [SD], 5.9 [1.00]), and complete (mean [SD], 5.8 [1.11]). Participants considered the tool easy to use (mean [SD], 5.4 [1.32]) and thought that it added value to the guideline enterprise (mean [SD], 5.9 [1.13]). Internal consistency of the items was high (Cronbach ??=?0.94). Positive correlations were found between the overall AGREE-REX score and the implementability score (r?=?0.81) and the clinical credibility score (r?=?0.76). Conclusions and Relevance:This cross-sectional study found that the AGREE-REX tool can be useful in evaluating CPG recommendations, differentiating among them, and identifying those that are clinically credible and implementable for practicing health professionals and decision makers who use recommendations to inform clinical policy.

Project description:BackgroundClinical networks are being used widely to facilitate large system transformation in healthcare, by engagement of stakeholders throughout the health system. However, there are no available instruments that measure engagement in these networks.MethodsThe study purpose was to develop and assess the measurement properties of a multiprofessional tool to measure engagement in clinical network initiatives. Based on components of the International Association of Public Participation Spectrum and expert panel review, we developed 40 items for testing. The draft instrument was distributed to 1,668 network stakeholders across different governance levels (leaders, members, support, frontline stakeholders) in 9 strategic clinical networks in Alberta (January to July 2014). With data from 424 completed surveys (25.4% response rate), descriptive statistics, exploratory and confirmatory factor analysis, Pearson correlations, linear regression, multivariate analysis, and Cronbach alpha were conducted to assess reliability and validity of the scores.ResultsSixteen items were retained in the instrument. Exploratory factor analysis indicated a four-factor solution and accounted for 85.7% of the total variance in engagement with clinical network initiatives: global engagement, inform (provided with information), involve (worked together to address concerns), and empower (given final decision-making authority). All subscales demonstrated acceptable reliability (Cronbach alpha 0.87 to 0.99). Both the confirmatory factor analysis and regression analysis confirmed that inform, involve, and empower were all significant predictors of global engagement, with involve as the strongest predictor. Leaders had higher mean scores than frontline stakeholders, while members and support staff did not differ in mean scores.ConclusionsThis study provided foundational evidence for the use of this tool for assessing engagement in clinical networks. Further work is necessary to evaluate engagement in broader network functions and activities; to assess barriers and facilitators of engagement; and, to elucidate how the maturity of networks and other factors influence engagement.

Project description:BackgroundResearch in 2007 showed that World Health Organization (WHO) recommendations were largely based on expert opinion, rarely used systematic evidence-based methods, and did not follow the organization's own "Guidelines for Guidelines". In response, the WHO established a "Guidelines Review Committee" (GRC) to implement and oversee internationally recognized standards. We examined the impact of these changes on WHO guideline documents and explored senior staff's perceptions of the new procedures.Methods and findingsWe used the AGREE II guideline appraisal tool to appraise ten GRC-approved guidelines from nine WHO departments, and ten pre-GRC guidelines matched by department and topic. We interviewed 20 senior staff across 16 departments and analyzed the transcripts using the framework approach. Average AGREE II scores for GRC-approved guidelines were higher across all six AGREE domains compared with pre-GRC guidelines. The biggest changes were noted for "Rigour of Development" (up 37.6%, from 30.7% to 68.3%) and "Editorial Independence" (up 52.7%, from 20.9% to 73.6%). Four main themes emerged from the interviews: (1) high standards were widely recognized as essential for WHO credibility, particularly with regard to conflicts of interest; (2) views were mixed on whether WHO needed a single quality assurance mechanism, with some departments purposefully bypassing the procedures; (3) staff expressed some uncertainties in applying the GRADE approach, with departmental staff concentrating on technicalities while the GRC remained concerned the underlying principles were not fully institutionalized; (4) the capacity to implement the new standards varied widely, with many departments looking to an overstretched GRC for technical support.ConclusionsSince 2007, WHO guideline development methods have become more systematic and transparent. However, some departments are bypassing the procedures, and as yet neither the GRC, nor the quality assurance standards they have set, are fully embedded within the organization.

Dataset Information

Efficient clinical evaluation of guideline quality: development and testing of a new tool.

Background

Methods

Results

Conclusion

Publications

Efficient clinical evaluation of guideline quality: development and testing of a new tool.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets