Dataset Information

Toward a comprehensive evidence map of overview of systematic review methods: paper 2-risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence.

ABSTRACT:

Background

Overviews of systematic reviews (SRs) attempt to systematically retrieve and summarise the results of multiple systematic reviews. This is the second of two papers from a study aiming to develop a comprehensive evidence map of the methods used in overviews. Our objectives were to (a) develop a framework of methods for conducting, interpreting and reporting overviews (stage I)-the Methods for Overviews of Reviews (MOoR) framework-and (b) to create an evidence map by mapping studies that have evaluated overview methods to the framework (stage II). In the first paper, we reported findings for the four initial steps of an overview (specification of purpose, objectives and scope; eligibility criteria; search methods; data extraction). In this paper, we report the remaining steps: assessing risk of bias; synthesis, presentation and summary of the findings; and assessing certainty of the evidence arising from the overview.

Methods

In stage I, we identified cross-sectional studies, guidance documents and commentaries that described methods proposed for, or used in, overviews. Based on these studies, we developed a framework of possible methods for overviews, categorised by the steps in conducting an overview. Multiple iterations of the framework were discussed and refined by all authors. In stage II, we identified studies evaluating methods and mapped these evaluations to the framework.

Results

Forty-two stage I studies described methods relevant to one or more of the latter steps of an overview. Six studies evaluating methods were included in stage II. These mapped to steps involving (i) the assessment of risk of bias (RoB) in SRs (two SRs and three primary studies, all reporting evaluation of RoB tools) and (ii) the synthesis, presentation and summary of the findings (one primary study evaluating methods for measuring overlap).

Conclusion

Many methods have been described for use in the latter steps in conducting an overview; however, evaluation and guidance for applying these methods is sparse. The exception is RoB assessment, for which a multitude of tools exist-several with sufficient evaluation and guidance to recommend their use. Evaluation of other methods is required to provide a comprehensive evidence map.

SUBMITTER: Lunny C

PROVIDER: S-EPMC6186052 | biostudies-literature | 2018 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Toward a comprehensive evidence map of overview of systematic review methods: paper 2-risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence.

Lunny Carole C Brennan Sue E SE McDonald Steve S McKenzie Joanne E JE

Systematic reviews 20181012 1

<h4>Background</h4>Overviews of systematic reviews (SRs) attempt to systematically retrieve and summarise the results of multiple systematic reviews. This is the second of two papers from a study aiming to develop a comprehensive evidence map of the methods used in overviews. Our objectives were to (a) develop a framework of methods for conducting, interpreting and reporting overviews (stage I)-the Methods for Overviews of Reviews (MOoR) framework-and (b) to create an evidence map by mapping stu ...[more]

PMID: 30314530

Similar Datasets

Project description:BackgroundOverviews of systematic reviews attempt to systematically retrieve and summarise the results of multiple systematic reviews. Methods for conducting, interpreting and reporting overviews are in their infancy. To date, there has been no evidence map of the methods used in overviews, thus making it difficult to determine the gaps and priorities for methods research. Our objectives were to develop and populate a comprehensive framework of methods for conducting, interpreting and reporting overviews (stage I) and to create an evidence map by mapping studies that have evaluated overview methods to the framework (stage II).MethodsWe searched methods collections (e.g. Cochrane Methodology Register, Meth4ReSyn library, AHRQ Effective Health Care Program) to identify eligible studies for both stages of this research. In stage I, cross-sectional studies, guidance documents and commentaries that described methods proposed for, or used in, overviews were used to develop and populate the framework of methods. Drafts and multiple iterations of the framework were discussed and refined by all authors. In stage II, we identified and described studies evaluating overview methods and mapped these evaluations to the framework.ResultsIn this paper, we present results for the four initial steps of conducting an overview: (a) specification of the purpose, objectives and scope, (b) specification of the eligibility criteria, (c) search methods and (d) data extraction. Twenty-nine studies mentioned or described methods relevant to one or more of these steps. In the developed framework, identified methods and approaches were grouped according to the steps an overview author would need to undertake. Fifteen studies evaluated identified methods, all of which mapped to the search methods step. These studies either reported the development and evaluation of a new search filter to retrieve systematic reviews or compared the performance of multiple filters.ConclusionGaps in the evaluation of methods were found for the majority of steps in the framework. More empirical studies are needed to evaluate the methods outlined and provide a comprehensive evidence map. The framework is useful for planning these evaluations and for planning methods required to deal with challenges that arise when conducting an overview.

Project description:BackgroundIonizing radiation is an established carcinogen, but risks from low-dose exposures are controversial. Since the Biological Effects of Ionizing Radiation VII review of the epidemiological data in 2006, many subsequent publications have reported excess cancer risks from low-dose exposures. Our aim was to systematically review these studies to assess the magnitude of the risk and whether the positive findings could be explained by biases.MethodsEligible studies had mean cumulative doses of less than 100 mGy, individualized dose estimates, risk estimates, and confidence intervals (CI) for the dose-response and were published in 2006-2017. We summarized the evidence for bias (dose error, confounding, outcome ascertainment) and its likely direction for each study. We tested whether the median excess relative risk (ERR) per unit dose equals zero and assessed the impact of excluding positive studies with potential bias away from the null. We performed a meta-analysis to quantify the ERR and assess consistency across studies for all solid cancers and leukemia.ResultsOf the 26 eligible studies, 8 concerned environmental, 4 medical, and 14 occupational exposure. For solid cancers, 16 of 22 studies reported positive ERRs per unit dose, and we rejected the hypothesis that the median ERR equals zero (P = .03). After exclusion of 4 positive studies with potential positive bias, 12 of 18 studies reported positive ERRs per unit dose (P = .12). For leukemia, 17 of 20 studies were positive, and we rejected the hypothesis that the median ERR per unit dose equals zero (P = .001), also after exclusion of 5 positive studies with potential positive bias (P = .02). For adulthood exposure, the meta-ERR at 100 mGy was 0.029 (95% CI = 0.011 to 0.047) for solid cancers and 0.16 (95% CI = 0.07 to 0.25) for leukemia. For childhood exposure, the meta-ERR at 100 mGy for leukemia was 2.84 (95% CI = 0.37 to 5.32); there were only two eligible studies of all solid cancers.ConclusionsOur systematic assessments in this monograph showed that these new epidemiological studies are characterized by several limitations, but only a few positive studies were potentially biased away from the null. After exclusion of these studies, the majority of studies still reported positive risk estimates. We therefore conclude that these new epidemiological studies directly support excess cancer risks from low-dose ionizing radiation. Furthermore, the magnitude of the cancer risks from these low-dose radiation exposures was statistically compatible with the radiation dose-related cancer risks of the atomic bomb survivors.

Project description:BackgroundBest formats for summarising and presenting evidence for use in clinical guideline development remain less well defined. We aimed to assess the effectiveness of different evidence summary formats to address this gap.MethodsHealthcare professionals attending a one-week Kenyan, national guideline development workshop were randomly allocated to receive evidence packaged in three different formats: systematic reviews (SRs) alone, systematic reviews with summary-of-findings tables, and 'graded-entry' formats (a 'front-end' summary and a contextually framed narrative report plus the SR). The influence of format on the proportion of correct responses to key clinical questions, the primary outcome, was assessed using a written test. The secondary outcome was a composite endpoint, measured on a 5-point scale, of the clarity of presentation and ease of locating the quality of evidence for critical neonatal outcomes. Interviews conducted within two months following completion of trial data collection explored panel members' views on the evidence summary formats and experiences with appraisal and use of research information.Results65 (93%) of 70 participants completed questions on the prespecified outcome measures. There were no differences between groups in the odds of correct responses to key clinical questions. 'Graded-entry' formats were associated with a higher mean composite score for clarity and accessibility of information about the quality of evidence for critical neonatal outcomes compared to systematic reviews alone (adjusted mean difference 0.52, 95% CI 0.06 to 0.99). There was no difference in the mean composite score between SR with SoF tables and SR alone. Findings from interviews with 16 panelists indicated that short narrative evidence reports were preferred for the improved clarity of information presentation and ease of use.ConclusionsOur findings suggest that 'graded-entry' evidence summary formats may improve clarity and accessibility of research evidence in clinical guideline development.Trial registrationControlled-Trials.com ISRCTN05154264.

Project description:BACKGROUND:The GRADE-CERQual (Confidence in Evidence from Reviews of Qualitative research) approach has been developed by the GRADE (Grading of Recommendations Assessment, Development and Evaluation) Working Group. The approach has been developed to support the use of findings from qualitative evidence syntheses in decision making, including guideline development and policy formulation. CERQual includes four components for assessing how much confidence to place in findings from reviews of qualitative research (also referred to as qualitative evidence syntheses): (1) methodological limitations, (2) coherence, (3) adequacy of data and (4) relevance. This paper is part of a series providing guidance on how to apply CERQual and focuses on making an overall assessment of confidence in a review finding and creating a CERQual Evidence Profile and a CERQual Summary of Qualitative Findings table. METHODS:We developed this guidance by examining the methods used by other GRADE approaches, gathering feedback from relevant research communities and developing consensus through project group meetings. We then piloted the guidance on several qualitative evidence syntheses before agreeing on the approach. RESULTS:Confidence in the evidence is an assessment of the extent to which a review finding is a reasonable representation of the phenomenon of interest. Creating a summary of each review finding and deciding whether or not CERQual should be used are important steps prior to assessing confidence. Confidence should be assessed for each review finding individually, based on the judgements made for each of the four CERQual components. Four levels are used to describe the overall assessment of confidence: high, moderate, low or very low. The overall CERQual assessment for each review finding should be explained in a CERQual Evidence Profile and Summary of Qualitative Findings table. CONCLUSIONS:Structuring and summarising review findings, assessing confidence in those findings using CERQual and creating a CERQual Evidence Profile and Summary of Qualitative Findings table should be essential components of undertaking qualitative evidence syntheses. This paper describes the end point of a CERQual assessment and should be read in conjunction with the other papers in the series that provide information on assessing individual CERQual components.

Project description:BackgroundThe GRADE-CERQual (Confidence in Evidence from Reviews of Qualitative research) approach has been developed by the GRADE (Grading of Recommendations Assessment, Development and Evaluation) Working Group. The approach has been developed to support the use of findings from qualitative evidence syntheses in decision-making, including guideline development and policy formulation. CERQual includes four components for assessing how much confidence to place in findings from reviews of qualitative research (also referred to as qualitative evidence syntheses): (1) methodological limitations, (2) coherence, (3) adequacy of data and (4) relevance. This paper is part of a series providing guidance on how to apply CERQual and focuses on a probable fifth component, dissemination bias. Given its exploratory nature, we are not yet able to provide guidance on applying this potential component of the CERQual approach. Instead, we focus on how dissemination bias might be conceptualised in the context of qualitative research and the potential impact dissemination bias might have on an overall assessment of confidence in a review finding. We also set out a proposed research agenda in this area.MethodsWe developed this paper by gathering feedback from relevant research communities, searching MEDLINE and Web of Science to identify and characterise the existing literature discussing or assessing dissemination bias in qualitative research and its wider implications, developing consensus through project group meetings, and conducting an online survey of the extent, awareness and perceptions of dissemination bias in qualitative research.ResultsWe have defined dissemination bias in qualitative research as a systematic distortion of the phenomenon of interest due to selective dissemination of studies or individual study findings. Dissemination bias is important for qualitative evidence syntheses as the selective dissemination of qualitative studies and/or study findings may distort our understanding of the phenomena that these syntheses aim to explore and thereby undermine our confidence in these findings. Dissemination bias has been extensively examined in the context of randomised controlled trials and systematic reviews of such studies. The effects of potential dissemination bias are formally considered, as publication bias, within the GRADE approach. However, the issue has received almost no attention in the context of qualitative research. Because of very limited understanding of dissemination bias and its potential impact on review findings in the context of qualitative evidence syntheses, this component is currently not included in the GRADE-CERQual approach.ConclusionsFurther research is needed to establish the extent and impacts of dissemination bias in qualitative research and the extent to which dissemination bias needs to be taken into account when we assess how much confidence we have in findings from qualitative evidence syntheses.

Dataset Information

Toward a comprehensive evidence map of overview of systematic review methods: paper 2-risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence.

Background

Methods

Results

Conclusion

Publications

Toward a comprehensive evidence map of overview of systematic review methods: paper 2-risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets