Dataset Information

Statistical multiplicity in systematic reviews of anaesthesia interventions: a quantification and comparison between Cochrane and non-Cochrane reviews.

ABSTRACT:

Background

Systematic reviews with meta-analyses often contain many statistical tests. This multiplicity may increase the risk of type I error. Few attempts have been made to address the problem of statistical multiplicity in systematic reviews. Before the implications are properly considered, the size of the issue deserves clarification. Because of the emphasis on bias evaluation and because of the editorial processes involved, Cochrane reviews may contain more multiplicity than their non-Cochrane counterparts. This study measured the quantity of statistical multiplicity present in a population of systematic reviews and aimed to assess whether this quantity is different in Cochrane and non-Cochrane reviews.

Methods/principal findings

We selected all the systematic reviews published by the Cochrane Anaesthesia Review Group containing a meta-analysis and matched them with comparable non-Cochrane reviews. We counted the number of statistical tests done in each systematic review. The median number of tests overall was 10 (interquartile range (IQR) 6 to 18). The median was 12 in Cochrane and 8 in non-Cochrane reviews (difference in medians 4 (95% confidence interval (CI) 2.0-19.0). The proportion that used an assessment of risk of bias as a reason for doing extra analyses was 42% in Cochrane and 28% in non-Cochrane reviews (difference in proportions 14% (95% CI -8 to 36). The issue of multiplicity was addressed in 6% of all the reviews.

Conclusion/significance

Statistical multiplicity in systematic reviews requires attention. We found more multiplicity in Cochrane reviews than in non-Cochrane reviews. Many of the reasons for the increase in multiplicity may well represent improved methodological approaches and greater transparency, but multiplicity may also cause an increased risk of spurious conclusions. Few systematic reviews, whether Cochrane or non-Cochrane, address the issue of multiplicity.

SUBMITTER: Imberger G

PROVIDER: S-EPMC3229598 | biostudies-literature | 2011

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Statistical multiplicity in systematic reviews of anaesthesia interventions: a quantification and comparison between Cochrane and non-Cochrane reviews.

Imberger Georgina G Vejlby Alexandra Damgaard AD Hansen Sara Bohnstedt SB Møller Ann M AM Wetterslev Jørn J

PloS one 20111202 12

<h4>Background</h4>Systematic reviews with meta-analyses often contain many statistical tests. This multiplicity may increase the risk of type I error. Few attempts have been made to address the problem of statistical multiplicity in systematic reviews. Before the implications are properly considered, the size of the issue deserves clarification. Because of the emphasis on bias evaluation and because of the editorial processes involved, Cochrane reviews may contain more multiplicity than their n ...[more]

PMID: 22164288

Similar Datasets

Project description:ObjectiveOsteoarthritis generates substantial health and socioeconomic burden, which is particularly marked in marginalized groups. It is imperative that practitioners have ready access to summaries of evidence-based interventions for osteoarthritis that incorporate equity considerations. Summaries of systematic reviews can provide this. The present study surveyed experts to prioritize a selection ofinterventions, from which equity focused summaries will be generated. Specifically, the prioritized interventions will be developed into Cochrane Evidence4Equity (E4E) summaries.MethodsTwenty-seven systematic reviews of OA interventions were found. From these, twenty-nine non-surgical treatments for osteoarthritis were identified, based on statistically significant findings for desired outcome variables or adverse events. Key findings from these studies were summarised and provided to 9 experts in the field of osteoarthritis.. Expert participants were asked to rate interventions based on feasibility, health system effects, universality, impact on inequities, and priority for translation into equity based E4E summaries. Expert participants were also encouraged to make comments to provide context for each rating. Free text responses were coded inductively and grouped into subthemes and themes.ResultsExpert participants rated the intervention home land-based exercise for knee OA highest for priority for translation into an E4E summaries, followed by the interventions individual land-based exercise for knee OA, class land-based exercise for knee OA, exercise for hand OA and land-based exercise for hip OA. Upon qualitative analysis of the expert participants' comments, fifteen subthemes were identified and grouped into three overall themes: (1) this intervention or an aspect of this intervention is unnecessary or unsafe; (2) this intervention or an aspect of this intervention may increase health inequities; and (3) experts noted difficulties completing rating exercise.ConclusionThe list of priority interventions and corresponding expert commentary generated information that will be used to direct and support knowledge translation efforts.

Project description:BackgroundSeveral factors are associated with an increased risk of preterm birth (PTB); therefore, various interventions might have the potential to influence it. Due to the large number of interventions that address PTB, the objective of this overview is to summarise evidence from Cochrane reviews regarding the effects and safety of these different interventions.MethodsWe conducted a systematic literature search in the Cochrane Database of Systematic Reviews. Included reviews should be based on randomised controlled trials comparing antenatal non-pharmacological and pharmacological interventions that directly or indirectly address PTB with placebo/no treatment or routine care in pregnant women at less than 37 completed weeks of gestation without signs of threatened preterm labour. We considered PTB at less than 37 completed weeks of gestation as the primary outcome.ResultsWe included 56 Cochrane systematic reviews. Three interventions increased PTB risk significantly. Twelve interventions led to a statistically significant lower incidence of PTBs. However, this reduction was mostly observed in defined at-risk subgroups of pregnant women. The remaining antenatal interventions failed to prove a significant effect on PTB?<?37 weeks, but some of them at least showed a positive effect in secondary outcomes (e.g., reduction in early PTBs). As an unintended result of this review, we identified 28 additional Cochrane reviews which intended to report on PTB?<?37 weeks, but were not able to find any RCTs reporting appropriate data.ConclusionsThe possible effects of a diverse range of interventions on PTB have been evaluated in Cochrane systematic reviews. Few interventions have been demonstrated to be effective and a small number have been found to be harmful. For around half of the interventions evaluated, the Cochrane review concluded that there was insufficient evidence to provide sound recommendations for clinical practice. No RCT evidence is available for a number of potentially relevant interventions.

Project description:Eating disorders are psychiatric conditions originated from and perpetuated by individual, family and sociocultural factors. The psychosocial approach to treatment and prevention of relapse is crucial. To present an overview of the scientific evidence on effectiveness of psychosocial interventions in treatment of eating disorders. All systematic reviews published by the Cochrane Database of Systematic Reviews - Cochrane Library on the topic were included. Afterwards, as from the least recent date of these reviews (2001), an additional search was conducted at PubMed with sensitive search strategy and with the same keywords used. A total of 101 primary studies and 30 systematic reviews (5 Cochrane systematic reviews), meta-analysis, guidelines or narrative reviews of literature were included. The main outcomes were: symptomatic remission, body image, cognitive distortion, psychiatric comorbidity, psychosocial functioning and patient satisfaction. The cognitive behavioral approach was the most effective treatment, especially for bulimia nervosa, binge eating disorder and the night eating syndrome. For anorexia nervosa, the family approach showed greater effectiveness. Other effective approaches were interpersonal psychotherapy, dialectic behavioral therapy, support therapy and self-help manuals. Moreover, there was an increasing number of preventive and promotional approaches that addressed individual, family and social risk factors, being promising for the development of positive self-image and self-efficacy. Further studies are required to evaluate the impact of multidisciplinary approaches on all eating disorders, as well as the cost-effectiveness of some effective modalities, such as the cognitive behavioral therapy. RESUMO Transtornos alimentares são doenças psiquiátricas originadas de e perpetuadas por fatores individuais, familiares e socioculturais. A abordagem psicossocial é essencial para o tratamento e a prevenção de recaídas. Apresentar uma visão geral das evidências científicas sobre a efetividade das intervenções psicossociais no tratamento de transtornos alimentares. Foram incluídas todas as revisões sistemáticas publicadas no Banco de Dados de Revisões Sistemáticas da Cochrane Library. Posteriormente, a partir da data menos recente destas revisões (2001), realizou-se uma busca adicional no PubMed, com estratégia de busca sensibilizada e com os mesmos descritores utilizados antes. No total, foram incluídos 101 estudos primários e 30 revisões sistemáticas (5 revisões sistemáticas da Cochrane), metanálises, diretrizes ou revisões narrativas da literatura. Os principais desfechos foram remissão de sintomas, imagem corporal, distorção cognitiva, comorbidade psiquiátrica, funcionamento psicossocial e satisfação do paciente. A abordagem cognitivo-comportamental foi o tratamento mais efetivo, principalmente para bulimia nervosa, transtorno da compulsão alimentar periódica e síndrome do comer noturno. Para anorexia nervosa, a abordagem familiar demonstrou maior efetividade. Outras abordagens efetivas foram psicoterapia interpessoal, terapia comportamental dialética, terapia de apoio e manuais de autoajuda. Além disso, houve um número crescente de abordagens preventivas e promocionais que contemplaram fatores de risco individuais, familiares e sociais, sendo promissoras para o desenvolvimento da autoimagem positiva e autoeficácia. São necessários mais estudos que avaliem o impacto de abordagens multidisciplinares em todos transtornos alimentares, além da relação custo-efetividade de algumas modalidades efetivas, como a terapia cognitivo-comportamental.

Project description:This research-on-research study describes efforts to develop non-Cochrane systematic reviews (SRs) by analyzing demographical and time-course collaborations between international institutions using protocols registered in the International Prospective Register of Systematic Reviews (PROSPERO) or published in scientific journals. We have published an a priori protocol to develop this study. Protocols published in scientific journals were searched using the MEDLINE and Embase databases; the query terms "Systematic review" [Title] AND "protocol" [Title] were searched from February 2011 to December 2017. Protocols registered at PROSPERO during the same period were obtained by web scraping all non-Cochrane records with a Python script. After excluding protocols that had a fulfillment or duplication rate of less than 90%, they were classified as published "only in PROSPERO", "only in journals", or in "journals and PROSPERO". Results of data and metadata extraction using text mining processes were curated by two reviewers. These Datasets and R scripts are freely available to facilitate reproducibility. We obtained 20,814 protocols of non-Cochrane SRs. While "unique protocols" by reviewers' institutions from 60 countries were the most frequent, a median of 6 (2-150) institutions from 130 different countries were involved in the preparation of "collaborative protocols". The highest Ranked countries involved in overall protocol production were the UK, the U.S., Australia, Brazil, China, Canada, the Netherlands, Germany, and Italy. Most protocols were registered only in PROSPERO. However, the number of protocols published in scientific journals (924) or in both PROSPERO and journals (807) has increased over the last three years. Syst Rev and BMJ Open published more than half of the total protocols. While the more productive countries were involved in "unique" and "collaborative protocols", less productive countries only participated in "collaborative protocols" that were mainly published in PROSPERO. Our results suggest that, although most countries were involved in solitary production of protocols for non-Cochrane SRs during the study period, it would be useful to develop new strategies to promote international collaborations, especially with less productive countries.

Project description:BACKGROUND:Consumers, clinicians, policymakers and researchers require high quality evidence to guide decision-making in child health. Though Cochrane systematic reviews (SRs) are a well-established source of evidence, little is known about the characteristics of non-Cochrane child-relevant SRs. To complement published descriptions of Cochrane SRs, we aimed to characterize the epidemiologic, methodological, and reporting qualities of non-Cochrane child-relevant SRs published in 2014. METHODS:English-language child-relevant SRs of quantitative primary research published outside the Cochrane Library in 2014 were eligible for this descriptive analysis. A research librarian searched MEDLINE, CINAHL, Web of Science, and PubMed in August 2015. A single reviewer screened articles for inclusion; a second verified the excluded studies. Reviewers extracted: general characteristics of the review; included study characteristics; methodological approaches. We performed univariate analyses and presented the findings narratively. RESULTS:We identified 1598 child-relevant SRs containing a median (IQR) 19 (11, 33) studies. These originated primarily from high-income countries (n?=?1247, 78.0%) and spanned 47 of the 53 Cochrane Review Groups. Most synthesized therapeutic (n?=?753, 47.1%) or epidemiologic (n?=?701, 43.8%) evidence. Though 39.3% (n?=?628) of SRs included evidence related to children only, few were published in pediatric-specific journals (n?=?283, 17.7%). Reporting quality seemed poor based on the items we assessed; few reviews mentioned an a-priori protocol (n?=?246, 15.4%) or registration (n?=?111, 6.9%), and only 23.4% (n?=?374) specified a primary outcome. Many SRs relied solely on evidence from non-RCTs (n?=?796, 49.8%). Less than two-thirds (n?=?953, 59.6%) appraised the quality of included studies and assessments of the certainty of the body of evidence were rare (n?=?102, 6.4%). CONCLUSIONS:Child-relevant Cochrane SRs are a known source of high quality evidence in pediatrics. There exists, however, an abundance of evidence from non-Cochrane SRs that may be complementary. Our findings show that high-quality non-Cochrane SRs may not be practical nor easy for knowledge users to find. Improvements are needed to ensure that evidence syntheses published outside of the Cochrane Library adhere to the high standard of conduct and reporting characteristic of Cochrane SRs.

Project description:BackgroundThe evaluation of imprecision is a key dimension of the grading of the confidence in the estimate. Grading of Recommendations Assessment, Development and Evaluation (GRADE) gives recommendations on how to downgrade evidence for imprecision, but authors vary in their use. Trial Sequential Analysis (TSA) has been advocated for a more reliable assessment of imprecision. We aimed to evaluate reporting of and adherence to GRADE and to compare the assessment of imprecision of intervention effects assessed by GRADE and TSA in Cochrane systematic reviews.MethodsIn this cross-sectional study, we included 100 Cochrane reviews irrespective of type of intervention with a key dichotomous outcome meta-analyzed and assessed by GRADE. The methods and results sections of each review were assessed for adequacy of imprecision evaluation. We re-analyzed imprecision following the GRADE Handbook and the TSA Manual.ResultsOverall, only 13.0% of reviews stated the criteria they applied to assess imprecision. The most common dimensions were the 95% width of the confidence intervals and the optimal information size. Review authors downgraded 48.0% of key outcomes due to imprecision. When imprecision was re-analyzed following the GRADE Handbook, 64% of outcomes were downgraded. Agreement between review authors' assessment and assessment by the authors of this study was moderate (kappa 0.43, 95% confidence interval [CI] 0.23 to 0.58). TSA downgraded 69.0% outcomes due to imprecision. Agreement between review authors' GRADE assessment and TSA, irrespective of downgrading levels, was moderate (kappa 0.43, 95% CI 0.21 to 0.57). Agreement between our GRADE assessment following the Handbook and TSA was substantial (kappa 0.66, 95% CI 0.49 to 0.79).ConclusionsIn a sample of Cochrane reviews, methods for assessing imprecision were rarely reported. GRADE according to Handbook guidelines and TSA led to more severe judgment of imprecision rather than GRADE adopted by reviews' authors. Cochrane initiatives to improve adherence to GRADE Handbook are warranted. TSA may transparently assist in such development.

Project description:BackgroundPrevious research on data extraction methods in systematic reviews has focused on single aspects of the process. We aimed to provide a deeper insight into these methods by analysing a current sample of reviews.MethodsWe included systematic reviews of health interventions in humans published in English. We analysed 75 Cochrane reviews from May and June 2020 and a random sample of non-Cochrane reviews published in the same period and retrieved from Medline. We linked reviews with protocols and study registrations. We collected information on preparing, piloting, and performing data extraction and on use of software to assist review conduct (automation tools). Data were extracted by one author, with 20% extracted in duplicate. Data were analysed descriptively.ResultsOf the 152 included reviews, 77 reported use of a standardized extraction form (51%); 42 provided information on the type of form used (28%); 24 on piloting (16%); 58 on what data was collected (38%); 133 on the extraction method (88%); 107 on resolving disagreements (70%); 103 on methods to obtain additional data or information (68%); 52 on procedures to avoid data errors (34%); and 47 on methods to deal with multiple study reports (31%). Items were more frequently reported in Cochrane than non-Cochrane reviews. The data extraction form used was published in 10 reviews (7%). Use of software was rarely reported except for statistical analysis software and use of RevMan and GRADEpro GDT in Cochrane reviews. Covidence was the most frequent automation tool used: 18 reviews used it for study selection (12%) and 9 for data extraction (6%).ConclusionsReporting of data extraction methods in systematic reviews is limited, especially in non-Cochrane reviews. This includes core items of data extraction such as methods used to manage disagreements. Few reviews currently use software to assist data extraction and review conduct. Our results can serve as a baseline to assess the uptake of such tools in future analyses.

Dataset Information

Statistical multiplicity in systematic reviews of anaesthesia interventions: a quantification and comparison between Cochrane and non-Cochrane reviews.

Background

Methods/principal findings

Conclusion/significance

Publications

Statistical multiplicity in systematic reviews of anaesthesia interventions: a quantification and comparison between Cochrane and non-Cochrane reviews.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets