Performance of Between-study Heterogeneity Measures in the Cochrane Library.
Ontology highlight
ABSTRACT:
SUBMITTER: Ma X
PROVIDER: S-EPMC6167168 | biostudies-literature | 2018 Nov
REPOSITORIES: biostudies-literature
Publications
Performance of Between-study Heterogeneity Measures in the Cochrane Library.
Ma Xiaoyue X Lin Lifeng L Qu Zhiyong Z Zhu Motao M Chu Haitao H
Epidemiology (Cambridge, Mass.) 20181101 6
The growth in comparative effectiveness research and evidence-based medicine has increased attention to systematic reviews and meta-analyses. Meta-analysis synthesizes and contrasts evidence from multiple independent studies to improve statistical efficiency and reduce bias. Assessing heterogeneity is critical for performing a meta-analysis and interpreting results. As a widely used heterogeneity measure, the I statistic quantifies the proportion of total variation across studies that is caused ...[more]
Similar Datasets
Project description:BackgroundHeterogeneity has a key role in meta-analysis methods and can greatly affect conclusions. However, true levels of heterogeneity are unknown and often researchers assume homogeneity. We aim to: a) investigate the prevalence of unobserved heterogeneity and the validity of the assumption of homogeneity; b) assess the performance of various meta-analysis methods; c) apply the findings to published meta-analyses.Methods and findingsWe accessed 57,397 meta-analyses, available in the Cochrane Library in August 2012. Using simulated data we assessed the performance of various meta-analysis methods in different scenarios. The prevalence of a zero heterogeneity estimate in the simulated scenarios was compared with that in the Cochrane data, to estimate the degree of unobserved heterogeneity in the latter. We re-analysed all meta-analyses using all methods and assessed the sensitivity of the statistical conclusions. Levels of unobserved heterogeneity in the Cochrane data appeared to be high, especially for small meta-analyses. A bootstrapped version of the DerSimonian-Laird approach performed best in both detecting heterogeneity and in returning more accurate overall effect estimates. Re-analysing all meta-analyses with this new method we found that in cases where heterogeneity had originally been detected but ignored, 17-20% of the statistical conclusions changed. Rates were much lower where the original analysis did not detect heterogeneity or took it into account, between 1% and 3%.ConclusionsWhen evidence for heterogeneity is lacking, standard practice is to assume homogeneity and apply a simpler fixed-effect meta-analysis. We find that assuming homogeneity often results in a misleading analysis, since heterogeneity is very likely present but undetected. Our new method represents a small improvement but the problem largely remains, especially for very small meta-analyses. One solution is to test the sensitivity of the meta-analysis conclusions to assumed moderate and large degrees of heterogeneity. Equally, whenever heterogeneity is detected, it should not be ignored.
| S-EPMC3724681 | biostudies-literature
Project description:Meta-analysis has become a widely used tool to combine results from independent studies. The collected studies are homogeneous if they share a common underlying true effect size; otherwise, they are heterogeneous. A fixed-effect model is customarily used when the studies are deemed homogeneous, while a random-effects model is used for heterogeneous studies. Assessing heterogeneity in meta-analysis is critical for model selection and decision making. Ideally, if heterogeneity is present, it should permeate the entire collection of studies, instead of being limited to a small number of outlying studies. Outliers can have great impact on conventional measures of heterogeneity and the conclusions of a meta-analysis. However, no widely accepted guidelines exist for handling outliers. This article proposes several new heterogeneity measures. In the presence of outliers, the proposed measures are less affected than the conventional ones. The performance of the proposed and conventional heterogeneity measures are compared theoretically, by studying their asymptotic properties, and empirically, using simulations and case studies.
| S-EPMC5106349 | biostudies-literature
Project description:BackgroundFinding duplicates is an important phase of systematic review. However, no consensus regarding the methods to find duplicates has been provided. This study aims to describe a pragmatic strategy of combining auto- and hand-searching duplicates in systematic review and to evaluate the prevalence and characteristics of duplicates.Methods and findingsLiteratures regarding portal vein thrombosis (PVT) and Budd-Chiari syndrome (BCS) were searched by the PubMed, EMBASE, and Cochrane library databases. Duplicates included one index paper and one or more redundant papers. They were divided into type-I (duplicates among different databases) and type-II (duplicate publications in different journals/issues) duplicates. For type-I duplicates, reference items were further compared between index and redundant papers. Of 10936 papers regarding PVT, 2399 and 1307 were identified as auto- and hand-searched duplicates, respectively. The prevalence of auto- and hand-searched redundant papers was 11.0% (1201/10936) and 6.1% (665/10936), respectively. They included 3431 type-I and 275 type-II duplicates. Of 11403 papers regarding BCS, 3275 and 2064 were identified as auto- and hand-searched duplicates, respectively. The prevalence of auto- and hand-searched redundant papers was 14.4% (1640/11403) and 9.1% (1039/11403), respectively. They included 5053 type-I and 286 type-II duplicates. Most of type-I duplicates were identified by auto-searching method (69.5%, 2385/3431 in PVT literatures; 64.6%, 3263/5053 in BCS literatures). Nearly all type-II duplicates were identified by hand-searching method (94.9%, 261/275 in PVT literatures; 95.8%, 274/286 in BCS literatures). Compared with those identified by auto-searching method, type-I duplicates identified by hand-searching method had a significantly higher prevalence of wrong items (47/2385 versus 498/1046, p<0.0001 in PVT literatures; 30/3263 versus 778/1790, p<0.0001 in BCS literatures). Most of wrong items originated from EMBASE database.ConclusionGiven the inadequacy of a single strategy of auto-searching method, a combined strategy of auto- and hand-searching methods should be employed to find duplicates in systematic review.
| S-EPMC3748039 | biostudies-literature
Project description:ObjectiveEvidence-based performance measures have been increasingly used to evaluate hospital quality of stroke care, but their impact on stroke outcomes has not been verified. We aimed to evaluate the correlations between hospital performance measures and outcomes among patients with acute ischaemic stroke in a Chinese population.MethodsData were derived from a prospective cohort, which included 120 hospitals participating in the China National Stroke Registry between September 2007 and August 2008. Adherence to nine evidence-based performance measures was examined, and the composite score of hospital performance measures was calculated. The primary stroke outcomes were hospital-level, 30-day and 1-year risk-standardised mortality (RSM). Associations of individual performance measures and composite score with stroke outcomes were assessed using Spearman correlation coefficients.ResultsOne hundred and twenty hospitals that recruited 12?027 patients with ischaemic stroke were included in our analysis. Among 12?027 patients, 61.59% were men, and the median age was 67 years. The overall composite score of performance measures was 63.3%. The correlation coefficients between individual performance measures ranged widely from 0.01 to 0.66. No association was observed between the composite score and 30-day RSM. The composite score was modestly associated with 1-year RSM (Spearman correlation coefficient, 0.34; p<0.05). The composite score explained only 2.53% and 10.18% of hospital-level variation in 30-day and 1-year RSM for patients with acute stroke.ConclusionsAdherence to evidence-based performance measures for acute ischaemic stroke was suboptimal in China. There were various correlations among hospital individual performance measures. The hospital performance measures had no correlations with 30-day RSM rate and modest correlations with 1-year RSM rate.
| S-EPMC6074631 | biostudies-literature
Project description:BackgroundEvidence-based decision making relies on easy access to trustworthy research results. The Cochrane Library is a key source of evidence about the effect of interventions and aims to "promote the accessibility of systematic reviews to anyone wanting to make a decision about health care". We explored how health professionals found, used and experienced The Library, looking at facets of user experience including findability, usability, usefulness, credibility, desirability and value.MethodsWe carried out 32 one-hour usability tests on participants from Norway and the UK. Participants both browsed freely and attempted to perform individually tailored tasks while "thinking aloud". Sessions were recorded and viewed in real time by researchers. Transcriptions and videos were reviewed by one researcher and one designer. Findings reported here reflect issues receiving a high degree of saturation and that we judge to be critical to the user experience of evidence-based web sites, based on principles for usability heuristics, web guidelines and evidence-based practice.ResultsParticipants had much difficulty locating both the site and its contents. Non-native English speakers were at an extra disadvantage when retrieving relevant documents despite high levels of English-language skills. Many participants displayed feelings of ineptitude, alienation and frustration. Some made serious mistakes in correctly distinguishing between different information types, for instance reviews, review protocols, and individual studies. Although most expressed a high regard for the site's credibility, some later displayed a mistrust of the independence of the information. Others were overconfident, thinking everything on The Cochrane Library site shared the same level of quality approval.ConclusionParadoxically, The Cochrane Library, established to support easy access to research evidence, has its own problems of accessibility. Health professionals' experiences of this and other evidence-based online resources can be improved by applying existing principles for web usability, prioritizing the development of simple search functionality, emitting "researcher" jargon, consistent marking of site ownership, and clear signposting of different document types and different content quality.
| S-EPMC2529276 | biostudies-literature
Project description:Meta-analyses conducted via the Cochrane Collaboration adhere to strict methodological and reporting standards aiming to minimize bias, maximize transparency/reproducibility, and improve the accuracy of summarized data. Whether this results in differences in the results reported by meta-analyses on the same topic conducted outside the Cochrane Collaboration is an open question.We conducted a matched-pair analysis with individual meta-analyses as the unit of analysis, comparing Cochrane and non-Cochrane reviews. Using meta-analyses from the cardiovascular literature, we identified pairs that matched on intervention and outcome. The pairs were contrasted in terms of how frequently results disagreed between the Cochrane and non-Cochrane reviews, whether effect sizes and statistical precision differed systematically, and how these differences related to the frequency of secondary citations of those reviews.Our search yielded 40 matched pairs of reviews. The two sets were similar in terms of which was first to publication, how many studies were included, and average sample sizes. The paired reviews included a total of 344 individual clinical trials: 111 (32.3%) studies were included only in a Cochrane review, 104 (30.2%) only in a non-Cochrane review, and 129 (37.5%) in both. Stated another way, 62.5% of studies were only included in one or the other meta-analytic literature. Overall, 37.5% of pairs had discrepant results. The most common involved shifts in the width of 95% confidence intervals that would yield a different statistical interpretation of the significance of results (7 pairs). Additionally, 20% differed in the direction of the summary effect size (5 pairs) or reported greater than a 2-fold difference in its magnitude (3 pairs). Non-Cochrane reviews reported significantly higher effect sizes (P< 0.001) and lower precision (P<0.001) than matched Cochrane reviews. Reviews reporting an effect size at least 2-fold greater than their matched pair were cited more frequently.Though results between topic-matched Cochrane and non-Cochrane reviews were quite similar, discrepant results were frequent, and the overlap of included studies was surprisingly low. Non-Cochrane reviews report larger effect sizes with lower precision than Cochrane reviews, indicating systematic differences, likely reflective of methodology, between the two types of reviews that could generate different interpretations of the interventions under question.
| S-EPMC4686011 | biostudies-literature
Project description:ObjectivesTo explore clinical and demographic characteristics impacting patient functioning by determining extent of overlap in factors driving change in Personal and Social Performance (PSP) and other clinical outcomes.MethodsPost-hoc analysis from a single-arm trial of paliperidone extended release in adult patients with nonacute symptomatic schizophrenia. Psychosocial functioning measures: PSP, Clinical Global Impression-Severity (CGI-S), Positive and Negative Syndrome Scale (PANSS), Short-Form 36 (SF-36), treatment satisfaction, sleep quality/daytime drowsiness, and Extrapyramidal Symptoms Rating Scale.ResultsHighest correlations with PSP total score change included PANSS total score change (Spearman's r = 0.607), PANSS general psychopathology change (r = 0.579), and CGI-S change (r = 0.569). A PSP score change of -32 predicted 90% probability of deterioration in CGI-S (score change of ≥1). The power of PSP change to predict PANSS total score change was lower. Linear stepwise regression demonstrated independent relationships for PSP change and: PANSS total change; CGI-S change; SF-36 Mental Component change; treatment satisfaction at endpoint; PSP at baseline; previous psychiatric hospitalizations. R 2 = 0.55 meant that 45% of PSP variation could not be explained by other clinical outcome measures.ConclusionsPsychosocial functioning improvement is important in schizophrenia. PSP may be valuable for assessing functioning; it encompasses psychosocial and clinical factors not measured by other established assessments.
| S-EPMC8170566 | biostudies-literature
Project description:BackgroundMuscle pump dysfunction is an essential component of chronic venous disease (CVD) pathology. Aging reduces muscle strength which further weakens the venous return. However, the epidemiology of CVD and its relationship with the physical performance in older persons is poorly studied. We studied the prevalence of CVD in subjects aged over 70 years and its association primarily with the Short Physical Performance Battery (SPPB) and 10 m walk test.MethodsAn accurate clinical leg examination was performed and the Clinical-Etiological-Anatomical-Pathophysiological-classification (CEAP, clinical classification of chronic venous disorders, C1-C6) determined by dermatologists in 552 subjects aged between 70 and 93 years belonging to the Northern Finland Birth Cohort 1966 - Parents' Study (NFBC-PS). Linear regression analyses were used to examine the association between CVD and functional tests and anthropometric measurements.ResultsThe prevalence of CVD (C1-C6) was 54.3%. C1 was diagnosed in 22.1% (n=84), C2 in 15.2% (n=45), C3 in 8.2% (n=45), C4 in 7.8% (43), C5 in 0.4% (n=2) and C6 in 0.5% (n=3). The prevalence and severity of CVD increased with increasing age (p<0.05). Males presented more with severe stages of CVD (C4-C6) (p<0.001). Subjects with CVD had significantly lower total SPPB scores and longer times in the 10 m walk test (p<0.001). The association between CVD severity and SPPB remained statistically significant in females after adjusting for age, body mass index (BMI) and number of children. The 10 m walk test times were associated with CVD when adjusted for sex and age but not after adjusting for BMI.ConclusionsIt is recommended that detailed skin examination of legs should be performed by physicians treating older subjects in order to improve early diagnosis of CVD. We highlight the importance of physical activity in older persons - lower limb activation of older persons with CVD may improve venous return and therefore prevent progression of CVD. We found an association between CVD and gait speed, however, there may exist bidirectional relationship.
| S-EPMC8518156 | biostudies-literature
Project description:BackgroundSystematic reviews with meta-analyses often contain many statistical tests. This multiplicity may increase the risk of type I error. Few attempts have been made to address the problem of statistical multiplicity in systematic reviews. Before the implications are properly considered, the size of the issue deserves clarification. Because of the emphasis on bias evaluation and because of the editorial processes involved, Cochrane reviews may contain more multiplicity than their non-Cochrane counterparts. This study measured the quantity of statistical multiplicity present in a population of systematic reviews and aimed to assess whether this quantity is different in Cochrane and non-Cochrane reviews.Methods/principal findingsWe selected all the systematic reviews published by the Cochrane Anaesthesia Review Group containing a meta-analysis and matched them with comparable non-Cochrane reviews. We counted the number of statistical tests done in each systematic review. The median number of tests overall was 10 (interquartile range (IQR) 6 to 18). The median was 12 in Cochrane and 8 in non-Cochrane reviews (difference in medians 4 (95% confidence interval (CI) 2.0-19.0). The proportion that used an assessment of risk of bias as a reason for doing extra analyses was 42% in Cochrane and 28% in non-Cochrane reviews (difference in proportions 14% (95% CI -8 to 36). The issue of multiplicity was addressed in 6% of all the reviews.Conclusion/significanceStatistical multiplicity in systematic reviews requires attention. We found more multiplicity in Cochrane reviews than in non-Cochrane reviews. Many of the reasons for the increase in multiplicity may well represent improved methodological approaches and greater transparency, but multiplicity may also cause an increased risk of spurious conclusions. Few systematic reviews, whether Cochrane or non-Cochrane, address the issue of multiplicity.
| S-EPMC3229598 | biostudies-literature
Project description:BACKGROUND: Randomized trials are essential in assessing the effects of healthcare interventions and are a key component in systematic reviews of effectiveness. Searching for reports of randomized trials in databases is problematic due to the absence of appropriate indexing terms until the 1990s and inconsistent application of these indexing terms thereafter. OBJECTIVES: The objectives of this study are to devise a search strategy for identifying reports of randomized trials in EMBASE which are not already indexed as trials in MEDLINE and to make these reports easily accessible by including them in the Cochrane Central Register of Controlled Trials (CENTRAL) in The Cochrane Library, with the permission of Elsevier, the publishers of EMBASE. METHODS: A highly sensitive search strategy was designed for EMBASE based on free-text and thesaurus terms which occurred frequently in the titles, abstracts, EMTREE terms (or some combination of these) of reports of trials indexed in EMBASE. This search strategy was run against EMBASE from 1980 to 2005 (1974 to 2005 for four of the terms) and records retrieved by the search, which were not already indexed as randomized trials in MEDLINE, were downloaded from EMBASE, printed and read. An analysis of the language of publication was conducted for the reports of trials published in 2005 (the most recent year completed at the time of this study). RESULTS: Twenty-two search terms were used (including nine which were later rejected due to poor cumulative precision). More than a third of a million records were downloaded and scanned and approximately 80,000 reports of trials were identified which were not already indexed as randomized trials in MEDLINE. These are now easily identifiable in CENTRAL, in The Cochrane Library. Cumulative sensitivity ranged from 0.1% to 60% and cumulative precision ranged from 8% to 61%. The truncated term 'random$' identified 60% of the total number of reports of trials but only 35% of the more than 130,000 records retrieved by this term were reports of trials. The language analysis for the sample year 2005 indicated that of the 18,427 reports indexed as randomized trials in MEDLINE, 959 (5%) were in languages other than English. The EMBASE search identified an additional 658 reports in languages other than English, of which the highest number were in Chinese (320). CONCLUSION: The results of the search to date have greatly increased access to reports of trials in EMBASE, especially in some languages other than English. The search strategy used was subjectively derived from a small 'gold standard' set of test records and was not validated in an independent test set. We intend to design an objectively-derived validated search strategy using logistic regression based on the frequency of occurrence of terms in the approximately 80,000 reports of randomized trials identified compared with the frequency of these terms across the entire EMBASE database.
| S-EPMC2586626 | biostudies-literature