Dataset Information

Systems for grading the quality of evidence and the strength of recommendations II: pilot study of a new system.

ABSTRACT:

Background

Systems that are used by different organisations to grade the quality of evidence and the strength of recommendations vary. They have different strengths and weaknesses. The GRADE Working Group has developed an approach that addresses key shortcomings in these systems. The aim of this study was to pilot test and further develop the GRADE approach to grading evidence and recommendations.

Methods

A GRADE evidence profile consists of two tables: a quality assessment and a summary of findings. Twelve evidence profiles were used in this pilot study. Each evidence profile was made based on information available in a systematic review. Seventeen people were given instructions and independently graded the level of evidence and strength of recommendation for each of the 12 evidence profiles. For each example judgements were collected, summarised and discussed in the group with the aim of improving the proposed grading system. Kappas were calculated as a measure of chance-corrected agreement for the quality of evidence for each outcome for each of the twelve evidence profiles. The seventeen judges were also asked about the ease of understanding and the sensibility of the approach. All of the judgements were recorded and disagreements discussed.

Results

There was a varied amount of agreement on the quality of evidence for the outcomes relating to each of the twelve questions (kappa coefficients for agreement beyond chance ranged from 0 to 0.82). However, there was fair agreement about the relative importance of each outcome. There was poor agreement about the balance of benefits and harms and recommendations. Most of the disagreements were easily resolved through discussion. In general we found the GRADE approach to be clear, understandable and sensible. Some modifications were made in the approach and it was agreed that more information was needed in the evidence profiles.

Conclusion

Judgements about evidence and recommendations are complex. Some subjectivity, especially regarding recommendations, is unavoidable. We believe our system for guiding these complex judgements appropriately balances the need for simplicity with the need for full and transparent consideration of all important issues.

SUBMITTER: Atkins D

PROVIDER: S-EPMC1084246 | biostudies-literature | 2005 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Systems for grading the quality of evidence and the strength of recommendations II: pilot study of a new system.

Atkins David D Briss Peter A PA Eccles Martin M Flottorp Signe S Guyatt Gordon H GH Harbour Robin T RT Hill Suzanne S Jaeschke Roman R Liberati Alessandro A Magrini Nicola N Mason James J O'Connell Dianne D Oxman Andrew D AD Phillips Bob B Schünemann Holger H Edejer Tessa Tan-Torres TT Vist Gunn E GE Williams John W JW

BMC health services research 20050323 1

<h4>Background</h4>Systems that are used by different organisations to grade the quality of evidence and the strength of recommendations vary. They have different strengths and weaknesses. The GRADE Working Group has developed an approach that addresses key shortcomings in these systems. The aim of this study was to pilot test and further develop the GRADE approach to grading evidence and recommendations.<h4>Methods</h4>A GRADE evidence profile consists of two tables: a quality assessment and a ...[more]

PMID: 15788089

Similar Datasets

Project description:BackgroundSignificant resources are required to write de novo clinical practice guidelines (CPGs). There are many freely-available CPGs internationally, for many health conditions. Developing countries rarely have the resources for de novo CPGs, and there could be efficiencies in using CPGs developed elsewhere. This paper outlines a novel process developed and tested in a resource-constrained country (South Africa) to synthesise findings from multiple international CPGs on allied health (AH) stroke rehabilitation.MethodsMethodologists, policy-makers, content experts and consumers collaborated to describe the pathway of an 'average' stroke patient through the South African public healthcare system and pose questions about best-practice stroke rehabilitation along this pathway. A comprehensive search identified international guidance documents published since January 2010. These were scanned for relevance to the South African AH stroke rehabilitation questions and critically appraised for methodological quality. Recommendations were extracted from guidance documents for each question. Strength of the body of evidence (SoBE) gradings underpinning recommendations were standardised, and composite recommendations were developed using qualitative synthesis. An algorithm was developed to guide assignment of overall SoBE gradings to composite recommendations.ResultsSixteen CPGs were identified, and all were included, as they answered different project questions differently. Methodological quality varied and was unrelated to currency. Seven clusters, outlining 20 composite recommendations were proposed (organise for best practice rehabilitation, operationalise strategies for best practice communication throughout the patient journey, admit to an acute hospital, refer to inpatient rehabilitation, action inpatient rehabilitation, discharge from inpatient rehabilitation and longer-term community-based rehabilitation).ConclusionThe methodological development process, tested by writing a South African AH stroke rehabilitation guideline from existing evidence sources, took 9 months. The process was efficient, collaborative, effective, rewarding and positive. Using the proposed methods, similar synthesis of existing evidence could be conducted in shorter time periods, in other resource-constrained countries, avoiding the need for expensive and time-consuming de novo CPG development.

Project description:BackgroundEvidence is necessary but not sufficient for decision-making, such as making recommendations by clinical practice guideline panels. However, the fundamental premise of evidence-based medicine (EBM) rests on the assumed link between the quality of evidence and "truth" and/or correctness in making guideline recommendations. If this assumption is accurate, then the quality of evidence ought to play a key role in making guideline recommendations. Surprisingly, and despite the widespread penetration of EBM in health care, there has been no empirical research to date investigating the impact of quality of evidence on the strength of recommendations made by guidelines panels.MethodsThe American Association of Blood Banking (AABB) has recently convened a 12 member panel to develop clinical practice guidelines (CPG) for the use of fresh-frozen plasma (FFP) for 6 different clinical indications. The panel was instructed that 4 factors should play a role in making recommendation: quality of evidence, uncertainty about the balance between desirable (benefits) and undesirable effects (harms), uncertainty or variability in values and preferences, and uncertainty about whether the intervention represents a wise use of resources (costs). Each member of the panel was asked to make his/her final judgments on the strength of recommendation and the overall quality of the body of evidence. "Voting" was anonymous and was based on the use of GRADE (Grading quality of evidence and strength of recommendations) system, which clearly distinguishes between quality of evidence and strength of recommendations.ResultsDespite the fact that many factors play role in formulating CPG recommendations, we show that when the quality of evidence is higher, the probability of making a strong recommendation for or against an intervention dramatically increases. Probability of making strong recommendation was 62% when evidence is "moderate", while it was only 23% and 13% when evidence was "low" or "very low", respectively.ConclusionWe report the first empirical evaluation of the relationship between quality of evidence pertinent to a clinical question and strength of the corresponding guideline recommendations. Understanding the relationship between quality of evidence and probability of making (strong) recommendation has profound implications for the science of quality measurement in health care.

Dataset Information

Systems for grading the quality of evidence and the strength of recommendations II: pilot study of a new system.

Background

Methods

Results

Conclusion

Publications

Systems for grading the quality of evidence and the strength of recommendations II: pilot study of a new system.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets