Dataset Information

An empirical investigation of the potential impact of selective inclusion of results in systematic reviews of interventions: study protocol.

ABSTRACT:

Background

Systematic reviewers may encounter a multiplicity of outcome data in the reports of randomised controlled trials included in the review (for example, multiple measurement instruments measuring the same outcome, multiple time points, and final and change from baseline values). The primary objectives of this study are to investigate in a cohort of systematic reviews of randomised controlled trials of interventions for rheumatoid arthritis, osteoarthritis, depressive disorders and anxiety disorders: (i) how often there is multiplicity of outcome data in trial reports; (ii) the association between selection of trial outcome data included in a meta-analysis and the magnitude and statistical significance of the trial result, and; (iii) the impact of the selection of outcome data on meta-analytic results.

Methods/design

Forty systematic reviews (20 Cochrane, 20 non-Cochrane) of RCTs published from January 2010 to January 2012 and indexed in the Cochrane Database of Systematic Reviews (CDSR) or PubMed will be randomly sampled. The first meta-analysis of a continuous outcome within each review will be included. From each review protocol (where available) and published review we will extract information regarding which types of outcome data were eligible for inclusion in the meta-analysis (for example, measurement instruments, time points, analyses). From the trial reports we will extract all outcome data that are compatible with the meta-analysis outcome as it is defined in the review and with the outcome data eligibility criteria and hierarchies in the review protocol. The association between selection of trial outcome data included in a meta-analysis and the magnitude and statistical significance of the trial result will be investigated. We will also investigate the impact of the selected trial result on the magnitude of the resulting meta-analytic effect estimates.

Discussion

The strengths of this empirical study are that our objectives and methods are pre-specified and transparent. The results may inform methods guidance for systematic review conduct and reporting, particularly for dealing with multiplicity of randomised controlled trial outcome data.

SUBMITTER: Page MJ

PROVIDER: S-EPMC3626625 | biostudies-literature | 2013 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

An empirical investigation of the potential impact of selective inclusion of results in systematic reviews of interventions: study protocol.

Page Matthew J MJ McKenzie Joanne E JE Green Sally E SE Forbes Andrew B AB

Systematic reviews 20130410

<h4>Background</h4>Systematic reviewers may encounter a multiplicity of outcome data in the reports of randomised controlled trials included in the review (for example, multiple measurement instruments measuring the same outcome, multiple time points, and final and change from baseline values). The primary objectives of this study are to investigate in a cohort of systematic reviews of randomised controlled trials of interventions for rheumatoid arthritis, osteoarthritis, depressive disorders an ...[more]

PMID: 23575367

Similar Datasets

Project description:To explore whether systematic reviewers selectively include trial effect estimates in meta-analyses when multiple are available, and what impact this may have on meta-analytic effects.Cross-sectional study.We randomly selected systematic reviews of interventions from 2 clinical specialties published between January 2010 and 2012. The first presented meta-analysis of a continuous outcome in each review was selected (index meta-analysis), and all trial effect estimates that were eligible for inclusion in the meta-analysis (eg, from multiple scales or time points) were extracted from trial reports.We calculated a statistic (the Potential Bias Index (PBI)) to quantify and test for evidence of selective inclusion. The PBI ranges from 0 to 1; values above or below 0.5 are suggestive of selective inclusion of effect estimates more or less favourable to the intervention, respectively. The impact of any potential selective inclusion was investigated by comparing the index meta-analytic standardised mean difference (SMD) to the median of a randomly constructed distribution of meta-analytic SMDs (representing the meta-analytic SMD expected when there is no selective inclusion).31 reviews (250 trials) were included. The estimated PBI was 0.57 (95% CI 0.50 to 0.63), suggesting that trial effect estimates that were more favourable to the intervention were included in meta-analyses slightly more often than expected under a process consistent with random selection; however, the 95% CI included the null hypothesis of no selective inclusion. Any potential selective inclusion did not have an important impact on the meta-analytic effects.There was no clear evidence that selective inclusion of trial effect estimates occurred in this sample of meta-analyses. Further research on selective inclusion in other clinical specialties is needed. To enable readers to assess the risk of selective inclusion bias, we recommend that systematic reviewers report the methods used to select effect estimates to include in meta-analyses.

Project description:BackgroundClinical trial results registries may contain relevant unpublished information. Our main aim was to investigate the potential impact of the inclusion of reports from industry results registries on systematic reviews (SRs).MethodsWe identified a sample of 150 eligible SRs in PubMed via backward selection. Eligible SRs investigated randomized controlled trials of drugs and included at least 2 bibliographic databases (original search date: 11/2009). We checked whether results registries of manufacturers and/or industry associations had also been searched. If not, we searched these registries for additional trials not considered in the SRs, as well as for additional data on trials already considered. We reanalysed the primary outcome and harm outcomes reported in the SRs and determined whether results had changed. A "change" was defined as either a new relevant result or a change in the statistical significance of an existing result. We performed a search update in 8/2013 and identified a sample of 20 eligible SRs to determine whether mandatory results registration from 9/2008 onwards in the public trial and results registry ClinicalTrials.gov had led to its inclusion as a standard information source in SRs, and whether the inclusion rate of industry results registries had changed.Results133 of the 150 SRs (89%) in the original analysis did not search industry results registries. For 23 (17%) of these SRs we found 25 additional trials and additional data on 31 trials already included in the SRs. This additional information was found for more than twice as many SRs of drugs approved from 2000 as approved beforehand. The inclusion of the additional trials and data yielded changes in existing results or the addition of new results for 6 of the 23 SRs. Of the 20 SRs retrieved in the search update, 8 considered ClinicalTrials.gov or a meta-registry linking to ClinicalTrials.gov, and 1 considered an industry results registry.ConclusionThe inclusion of industry and public results registries as an information source in SRs is still insufficient and may result in publication and outcome reporting bias. In addition to an essential search in ClinicalTrials.gov, authors of SRs should consider searching industry results registries.

Project description:BackgroundOverviews of reviews (overviews) compile information from multiple systematic reviews (SRs) to provide a single synthesis of relevant evidence for decision-making. Overviews may identify multiple SRs that examine the same intervention for the same condition and include some, but not all, of the same primary studies. There is currently limited guidance on whether and how to include these overlapping SRs in overviews. Our objectives were to assess how different inclusion decisions in overviews of healthcare interventions affect their comprehensiveness and results, and document challenges encountered when making different inclusion decisions in overviews.MethodsWe used five inclusion decisions to conduct overviews across seven topic areas, resulting in 35 overviews. The inclusion decisions were (1) include all Cochrane and non-Cochrane SRs, (2) include only Cochrane SRs, or consider all Cochrane and non-Cochrane SRs but include only non-overlapping SRs, and in the case of overlapping SRs, select (3) the Cochrane SR, (4) the most recent SR (by publication or search date), or (5) the highest quality SR (assessed using AMSTAR). For each topic area and inclusion scenario, we documented the amount of outcome data lost and changed and the challenges involved.ResultsWhen conducting overviews, including only Cochrane SRs, instead of all SRs, often led to loss/change of outcome data (median 31% of outcomes lost/changed; range 0-100%). Considering all Cochrane and non-Cochrane SRs but including only non-overlapping SRs and selecting the Cochrane SR for groups of overlapping SRs (instead of the most recent or highest quality SRs) allowed the most outcome data to be recaptured (median 42% of lost/changed outcome recaptured; range 28-86%). Across all inclusion scenarios, challenges were encountered when extracting data from overlapping SRs.ConclusionsOverlapping SRs present a methodological challenge for overview authors. This study demonstrates that different inclusion decisions affect the comprehensiveness and results of overviews in different ways, depending in part on whether Cochrane SRs examine all intervention comparisons relevant to the overview. Study results were used to develop an evidence-based decision tool that provides practical guidance for overview authors.

Project description:BackgroundMobile phone usage has been rapidly increasing worldwide. mHealth could efficiently deliver high-quality health care, but the evidence supporting its current effectiveness is still mixed.ObjectiveWe performed a systematic review of systematic reviews to assess the impact or effectiveness of mobile health (mHealth) interventions in different health conditions and in the processes of health care service delivery.MethodsWe used a common search strategy of five major scientific databases, restricting the search by publication date, language, and parameters in methodology and content. Methodological quality was evaluated using the Measurement Tool to Assess Systematic Reviews (AMSTAR) checklist.ResultsThe searches resulted in a total of 10,689 articles. Of these, 23 systematic reviews (371 studies; more than 79,665 patients) were included. Seventeen reviews included studies performed in low- and middle-income countries. The studies used diverse mHealth interventions, most frequently text messaging (short message service, SMS) applied to different purposes (reminder, alert, education, motivation, prevention). Ten reviews were rated as low quality (AMSTAR score 0-4), seven were rated as moderate quality (AMSTAR score 5-8), and six were categorized as high quality (AMSTAR score 9-11). A beneficial impact of mHealth was observed in chronic disease management, showing improvement in symptoms and peak flow variability in asthma patients, reducing hospitalizations and improving forced expiratory volume in 1 second; improving chronic pulmonary diseases symptoms; improving heart failure symptoms, reducing deaths and hospitalization; improving glycemic control in diabetes patients; improving blood pressure in hypertensive patients; and reducing weight in overweight and obese patients. Studies also showed a positive impact of SMS reminders in improving attendance rates, with a similar impact to phone call reminders at reduced cost, and improved adherence to tuberculosis and human immunodeficiency virus therapy in some scenarios, with evidence of decrease of viral load.ConclusionsAlthough mHealth is growing in popularity, the evidence for efficacy is still limited. In general, the methodological quality of the studies included in the systematic reviews is low. For some fields, its impact is not evident, the results are mixed, or no long-term studies exist. Exceptions include the moderate quality evidence of improvement in asthma patients, attendance rates, and increased smoking abstinence rates. Most studies were performed in high-income countries, implying that mHealth is still at an early stage of development in low-income countries.

Project description:BACKGROUND: Worldwide, there are an estimated 15 million individuals with drug use disorders and over five times as many with alcohol use disorders (WHO 1:2, 2005). Most individuals with substance misuse have families who are affected. Initial scoping searches identified an expanse of broad and disparate studies and reviews on the family interventions for substance misuse. This systematic review of systematic reviews aims to bring together the expanse of research on the effectiveness of family-based interventions in substance misuse.Initial scoping searches identified an expanse of broad and disparate studies and reviews on the family interventions for substance misuse. This systematic review of systematic reviews aims to bring together the expanse of research on the effectiveness of family-based interventions in substance misuse. METHODS: Extensive electronic and manual searches will be undertaken. Screening, data extraction and quality assessment will be undertaken by two reviewers with disagreements resolved through discussion.The inclusion criteria will be that the study is a systematically undertaken review, the population is individuals with substance misuse problems and the interventions include a family-focused component. Reviews that focus on prevention rather than treatment will be excluded. The reviews will be assessed for quality and relevance. The evidence from included systematic reviews will be mapped by focus of intervention (promoting engagement of user into treatment/joint involvement in treatment of user/treating family member in own right) for both adults and adolescents for drug and/or alcohol misusers to allow assessment of the density of available evidence. The higher-quality, up-to-date evidence for each domain will be identified and described, and conclusions will be drawn with limitations of the evidence highlighted. DISCUSSION: This systematic review of systematic reviews will be an efficient and robust way of looking at the current state of the evidence in the field of family-based interventions for substance misuse. It will evaluate all the available systematic-review-level literature to report on the effectiveness of family-based psychological interventions in improving substance-related outcomes and improving health and wellbeing of substance misusers and/or their families. This will inform future treatment policies and commissioning decisions.In addition, it will identify areas of poor quality, inconsistency and gaps in the evidence base for family-based psychological interventions in substance misuse with respect to secondary evidence in order to inform future research. SYSTEMATIC REVIEW REGISTRATION: PROSPERO CRD42014006834.

Project description:IntroductionDry eye is a leading cause of ocular morbidity and economic and societal burden for patients and healthcare systems. There are several treatment options available for dry eye and high-quality systematic reviews synthesise the evidence for their effectiveness and potential harms.Methods and analysisWe will search the Cochrane Eyes and Vision US satellite (CEV@US) database of eyes and vision systematic reviews for systematic reviews on interventions for dry eye. CEV@US conducted an initial search of PubMed and Embase to populate the CEV@US database of eyes and vision systematic reviews in 2007, which was updated most recently in August 2021. We will search the database for systematic reviews published since 1 January 2016 because systematic reviews more than 5 years are unlikely to be up to date. We will consider Cochrane and non-Cochrane systematic reviews eligible for inclusion. Two authors will independently screen articles. We will include studies that evaluate interventions for dry eye and/or meibomian gland dysfunction with no restriction on types of participants or review language. We will select reliable systematic reviews (ie, those meeting pre-established methodological criteria) for inclusion, assessed by one investigator and verified by a second investigator. We will extract ratings of the certainty of evidence from within each review. We will report the degree of overlap for systematic reviews that answer similar questions and include overlapping primary studies. We will present results of the overview in alignment with guidelines in the Cochrane Handbook of Systematic Reviews of Interventions (Online Chapter 5: Overviews of Reviews), the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement, and an overview of reviews quality and transparency checklist. The anticipated start and completion dates for this overview are 1 May 2021 and 30 April 2022, respectively.Ethics and disseminationThis overview will not require the approval of an Ethics Committee because it will use published studies. We will publish results in a peer-reviewed journal.Prospero registration numberCRD42021279880.

Project description:BackgroundBefore implementing healthcare interventions, clinicians need to weigh the beneficial and adverse effects of interventions. However, a large body of evidence has demonstrated that seeking and reporting of adverse effects is suboptimal in clinical trials and in systematic reviews of interventions. This cross-sectional study will investigate the status of this problem in orthodontics. This study will assess whether adverse effects were sought and whether findings related to adverse effects were reported in systematic reviews of orthodontic interventions in the five leading orthodontic journals and in the Cochrane Database of Systematic Reviews.MethodsSystematic reviews of clinical orthodontic interventions published between 01 August 2009 and 31 July 2019 in the five leading orthodontic journals and in the Cochrane Database will be included. Empty reviews will be excluded. The reporting of outcomes on adverse effects will not determine eligibility, i.e., reviews will not be excluded, because they did not report usable data. Study selection and data extraction will be conducted independently by two authors. Our primary outcome will be the prevalence of systematic reviews of orthodontic interventions that sought any findings related to adverse effects in the included studies. Additional prevalence statistics will be calculated on a series of items related to seeking of adverse effects in the eligible reviews. All statistics will be calculated for (1) all journals together, (2) the group of five orthodontic journals and the Cochrane Database of Systematic Reviews separately, and (3) each individual journal separately. Chi-square tests of independence will be used to compare these groups.DiscussionThis study will assess whether adverse effects were sought in systematic reviews of orthodontic interventions. This knowledge is important, because reviews that present an incomplete picture on adverse effects can have unfavorable consequences for the end-users. Also not reporting that no adverse effects were assessed in eligible studies included in a systematic review can mislead pertinent stakeholders. Our findings could have policy implications for making judgments on accepting or rejecting an intervention systematic review for publication, for example, by directing editors and peer-reviewers to adopt the various items on adverse effects defined in the MECIR standards and in the PRISMA harm checklist.

Dataset Information

An empirical investigation of the potential impact of selective inclusion of results in systematic reviews of interventions: study protocol.

Background

Methods/design

Discussion

Publications

An empirical investigation of the potential impact of selective inclusion of results in systematic reviews of interventions: study protocol.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets