Dataset Information

Impact of quality of evidence on the strength of recommendations: an empirical study.

ABSTRACT:

Background

Evidence is necessary but not sufficient for decision-making, such as making recommendations by clinical practice guideline panels. However, the fundamental premise of evidence-based medicine (EBM) rests on the assumed link between the quality of evidence and "truth" and/or correctness in making guideline recommendations. If this assumption is accurate, then the quality of evidence ought to play a key role in making guideline recommendations. Surprisingly, and despite the widespread penetration of EBM in health care, there has been no empirical research to date investigating the impact of quality of evidence on the strength of recommendations made by guidelines panels.

Methods

The American Association of Blood Banking (AABB) has recently convened a 12 member panel to develop clinical practice guidelines (CPG) for the use of fresh-frozen plasma (FFP) for 6 different clinical indications. The panel was instructed that 4 factors should play a role in making recommendation: quality of evidence, uncertainty about the balance between desirable (benefits) and undesirable effects (harms), uncertainty or variability in values and preferences, and uncertainty about whether the intervention represents a wise use of resources (costs). Each member of the panel was asked to make his/her final judgments on the strength of recommendation and the overall quality of the body of evidence. "Voting" was anonymous and was based on the use of GRADE (Grading quality of evidence and strength of recommendations) system, which clearly distinguishes between quality of evidence and strength of recommendations.

Results

Despite the fact that many factors play role in formulating CPG recommendations, we show that when the quality of evidence is higher, the probability of making a strong recommendation for or against an intervention dramatically increases. Probability of making strong recommendation was 62% when evidence is "moderate", while it was only 23% and 13% when evidence was "low" or "very low", respectively.

Conclusion

We report the first empirical evaluation of the relationship between quality of evidence pertinent to a clinical question and strength of the corresponding guideline recommendations. Understanding the relationship between quality of evidence and probability of making (strong) recommendation has profound implications for the science of quality measurement in health care.

SUBMITTER: Djulbegovic B

PROVIDER: S-EPMC2722589 | biostudies-literature | 2009 Jul

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Impact of quality of evidence on the strength of recommendations: an empirical study.

Djulbegovic Benjamin B Trikalinos Thomas A TA Roback John J Chen Ren R Guyatt Gordon G

BMC health services research 20090721

<h4>Background</h4>Evidence is necessary but not sufficient for decision-making, such as making recommendations by clinical practice guideline panels. However, the fundamental premise of evidence-based medicine (EBM) rests on the assumed link between the quality of evidence and "truth" and/or correctness in making guideline recommendations. If this assumption is accurate, then the quality of evidence ought to play a key role in making guideline recommendations. Surprisingly, and despite the wide ...[more]

PMID: 19622148

Similar Datasets

Project description:BackgroundSystems that are used by different organisations to grade the quality of evidence and the strength of recommendations vary. They have different strengths and weaknesses. The GRADE Working Group has developed an approach that addresses key shortcomings in these systems. The aim of this study was to pilot test and further develop the GRADE approach to grading evidence and recommendations.MethodsA GRADE evidence profile consists of two tables: a quality assessment and a summary of findings. Twelve evidence profiles were used in this pilot study. Each evidence profile was made based on information available in a systematic review. Seventeen people were given instructions and independently graded the level of evidence and strength of recommendation for each of the 12 evidence profiles. For each example judgements were collected, summarised and discussed in the group with the aim of improving the proposed grading system. Kappas were calculated as a measure of chance-corrected agreement for the quality of evidence for each outcome for each of the twelve evidence profiles. The seventeen judges were also asked about the ease of understanding and the sensibility of the approach. All of the judgements were recorded and disagreements discussed.ResultsThere was a varied amount of agreement on the quality of evidence for the outcomes relating to each of the twelve questions (kappa coefficients for agreement beyond chance ranged from 0 to 0.82). However, there was fair agreement about the relative importance of each outcome. There was poor agreement about the balance of benefits and harms and recommendations. Most of the disagreements were easily resolved through discussion. In general we found the GRADE approach to be clear, understandable and sensible. Some modifications were made in the approach and it was agreed that more information was needed in the evidence profiles.ConclusionJudgements about evidence and recommendations are complex. Some subjectivity, especially regarding recommendations, is unavoidable. We believe our system for guiding these complex judgements appropriately balances the need for simplicity with the need for full and transparent consideration of all important issues.

Project description:BackgroundSignificant resources are required to write de novo clinical practice guidelines (CPGs). There are many freely-available CPGs internationally, for many health conditions. Developing countries rarely have the resources for de novo CPGs, and there could be efficiencies in using CPGs developed elsewhere. This paper outlines a novel process developed and tested in a resource-constrained country (South Africa) to synthesise findings from multiple international CPGs on allied health (AH) stroke rehabilitation.MethodsMethodologists, policy-makers, content experts and consumers collaborated to describe the pathway of an 'average' stroke patient through the South African public healthcare system and pose questions about best-practice stroke rehabilitation along this pathway. A comprehensive search identified international guidance documents published since January 2010. These were scanned for relevance to the South African AH stroke rehabilitation questions and critically appraised for methodological quality. Recommendations were extracted from guidance documents for each question. Strength of the body of evidence (SoBE) gradings underpinning recommendations were standardised, and composite recommendations were developed using qualitative synthesis. An algorithm was developed to guide assignment of overall SoBE gradings to composite recommendations.ResultsSixteen CPGs were identified, and all were included, as they answered different project questions differently. Methodological quality varied and was unrelated to currency. Seven clusters, outlining 20 composite recommendations were proposed (organise for best practice rehabilitation, operationalise strategies for best practice communication throughout the patient journey, admit to an acute hospital, refer to inpatient rehabilitation, action inpatient rehabilitation, discharge from inpatient rehabilitation and longer-term community-based rehabilitation).ConclusionThe methodological development process, tested by writing a South African AH stroke rehabilitation guideline from existing evidence sources, took 9 months. The process was efficient, collaborative, effective, rewarding and positive. Using the proposed methods, similar synthesis of existing evidence could be conducted in shorter time periods, in other resource-constrained countries, avoiding the need for expensive and time-consuming de novo CPG development.

Project description:ObjectiveTo investigate whether alignment of strength of recommendations with quality of evidence differs in consensus based versus evidence based guidelines.DesignEmpirical analysis.Data sourceGuidelines developed by the American College of Cardiology and the American Heart Association (ACC/AHA) and the American Society of Clinical Oncology (ASCO) up to 27 March 2021.Study selectionRecommendations were clearly categorised as consensus or evidence based, were separated from the remainder of the text, and included both the quality of evidence and the strength of the recommendations.Data extractionPaired authors independently extracted the recommendation characteristics, including type of recommendation (consensus or evidence based), grading system used for developing recommendations, strength of the recommendation, and quality of evidence. The study team also calculated the number of discordant recommendations (strong recommendations with low quality evidence) and inappropriate discordant recommendations (those that did not meet grading of recommendations assessment, development, and evaluation criteria of appropriateness).ResultsThe study included 12 ACC/AHA guidelines that generated 1434 recommendations and 69 ASCO guidelines that generated 1094 recommendations. Of the 504 ACC/AHA recommendations based on low quality evidence, 200 (40%) proved to be consensus based versus 304 (60%) evidence based; of the 404 ASCO recommendations based on low quality evidence, 292 (72%) were consensus based versus 112 (28%) that were evidence based. In both ACC/AHA and ASCO guidelines, the consensus approach yielded more discordant recommendations (ACC/AHA: odds ratio 2.1, 95% confidence interval 1.5 to 3.1; ASCO: 2.9, 1.1 to 7.8) and inappropriate discordant recommendations (ACC/AHA: 2.6, 1.7 to 3.7; ASCO: 5.1, 1.6 to 16.0) than the evidence based approach.ConclusionConsensus based guidelines produce more recommendations violating the evidence based medicine principles than evidence based guidelines. Ensuring appropriate alignment of quality of evidence with the strength of recommendations is key to the development of "trustworthy" guidelines.

Project description:OBJECTIVES:To empirically assess the relation between study characteristics and prognostic model performance in external validation studies of multivariable prognostic models. DESIGN:Meta-epidemiological study. DATA SOURCES AND STUDY SELECTION:On 16 October 2018, we searched electronic databases for systematic reviews of prognostic models. Reviews from non-overlapping clinical fields were selected if they reported common performance measures (either the concordance (c)-statistic or the ratio of observed over expected number of events (OE ratio)) from 10 or more validations of the same prognostic model. DATA EXTRACTION AND ANALYSES:Study design features, population characteristics, methods of predictor and outcome assessment, and the aforementioned performance measures were extracted from the included external validation studies. Random effects meta-regression was used to quantify the association between the study characteristics and model performance. RESULTS:We included 10 systematic reviews, describing a total of 224 external validations, of which 221 reported c-statistics and 124 OE ratios. Associations between study characteristics and model performance were heterogeneous across systematic reviews. C-statistics were most associated with variation in population characteristics, outcome definitions and measurement and predictor substitution. For example, validations with eligibility criteria comparable to the development study were associated with higher c-statistics compared with narrower criteria (difference in logit c-statistic 0.21(95% CI 0.07 to 0.35), similar to an increase from 0.70 to 0.74). Using a case-control design was associated with higher OE ratios, compared with using data from a cohort (difference in log OE ratio 0.97(95% CI 0.38 to 1.55), similar to an increase in OE ratio from 1.00 to 2.63). CONCLUSIONS:Variation in performance of prognostic models across studies is mainly associated with variation in case-mix, study designs, outcome definitions and measurement methods and predictor substitution. Researchers developing and validating prognostic models should realise the potential influence of these study characteristics on the predictive performance of prognostic models.

Project description:I study the co-evolution between public opinion and party policy in situations of crises by investigating a policy U-turn of a major Austrian right-wing party (FPÖ) during the Covid-19 pandemic. My analysis suggests the existence of both i) a “Downsian” effect, which causes voters to adapt their party preferences based on policy congruence and ii) a “party identification” effect, which causes partisans to realign their policy preferences based on “their” party's platform. Specifically, I use individual-level panel data to show that i) “corona skeptical” voters who did not vote for the FPÖ in the pre-Covid-19 elections of 2019 were more likely to vote for the party after it embraced “corona populism”, and ii) beliefs of respondents who declared that they voted for the FPÖ in 2019 diverged from the rest of the population in three out of four health-related dimensions only after the turn, causing them to underestimate the threat posed by Covid-19 compared to the rest of the population. Using aggregate-level panel data, I study whether the turn has produced significant behavioral differences which could be observed in terms of reported cases and deaths per capita. Paradoxically, after the turn the FPÖ vote share is significantly positively correlated with deaths per capita, but not with the reported number of infections. I hypothesize that this can be traced back to a self-selection bias in testing, which causes a correlation between the number of “corona skeptics” and the share of unreported cases after the turn. I find empirical support for this hypothesis in individual-level data from a Covid-19 prevalence study that involves information about participants’ true vs. reported infection status. I finally study a simple heterogeneous mixing epidemiological model and show that a testing bias can indeed explain the apparent paradox of an increase in deaths without an increase in reported cases. My results can, among others, be used to enrich formal analyses regarding the co-evolution between voter and party behavior.

Project description:IntroductionThe rapid spread of drug resistance is forcing standard treatment guidelines (STGs) to become more appropriate with due consideration of the evidence on the antimicrobial resistance (AMR) situation in Ethiopia. Therefore, we aimed to assess the local AMR recommendations in the STGs for empirical antibacterial prescriptions for the five common infectious syndromes. We also determined the quality of AMR reviews conducted in the country.MethodsWe conducted a review of the STGs used in the health centers, general hospitals, and primary hospitals in Ethiopia and assessed the AMR recommendations in STGs for empirical antibacterial prescriptions for community-acquired pneumonia (CAP), urinary tract infection (UTI), tonsillopharyngitis, acute otitis media (AOM), and bacterial dysentery. Next, we performed an overview of AMR reviews published in Ethiopia. We used the MEDLINE/PubMed, Embase, Cochrane Library, and Google Scholar databases to identify AMR reviews. The review characteristics were extracted. We also evaluated the quality of each included AMR review using a measurement tool to assess the systematic review scale (AMSTAR 2).ResultsA total of 6 STGs and 12 AMR reviews conducted in the country were included. The choice of empirical antibacterials for similar infectious syndromes (and editions) was comparable across the three levels of the health care system. None of the STGs evaluated included the local AMR recommendations for empirical antibacterial prescriptions for five common infectious syndromes. Of all the AMR reviews included, 75% had low and below methodologic quality, and none had a high-quality score using the AMSTAR 2 tool.ConclusionStandard treatment guidelines did not consider local AMR recommendations for empirically prescribing antibacterials for common infectious syndromes. The AMR reviews published in the country produced poor methodologic quality evidence for clinical applications. This highlights the need to improve the methodologic quality to provide the best available evidence for clinical decision-making and curb the ongoing AMR in Ethiopia.Trial registrationRetrospectively registered (15/07/2020).

Dataset Information

Impact of quality of evidence on the strength of recommendations: an empirical study.

Background

Methods

Results

Conclusion

Publications

Impact of quality of evidence on the strength of recommendations: an empirical study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets