Dataset Information

Evaluating agreement between bodies of evidence from randomised controlled trials and cohort studies in nutrition research: meta-epidemiological study.

ABSTRACT:

Objective

To evaluate the agreement between diet-disease effect estimates of bodies of evidence from randomised controlled trials and those from cohort studies in nutrition research, and to investigate potential factors for disagreement.

Design

Meta-epidemiological study.

Data sources

Cochrane Database of Systematic Reviews, and Medline.

Review methods

Population, intervention or exposure, comparator, outcome (PI/ECO) elements from a body of evidence from cohort studies (BoE(CS)) were matched with corresponding elements of a body of evidence from randomised controlled trials (BoE(RCT)). Pooled ratio of risk ratios or difference of mean differences across all diet-disease outcome pairs were calculated. Subgroup analyses were conducted to explore factors for disagreement. Heterogeneity was assessed through I² and τ². Prediction intervals were calculated to assess the range of possible values for the difference in the results between evidence from randomised controlled trials and evidence from cohort studies in future comparisons.

Results

97 diet-disease outcome pairs (that is, matched BoE(RCT) and BoE(CS)) were identified overall. For binary outcomes, the pooled ratio of risk ratios comparing estimates from BoE(RCT) with BoE(CS) was 1.09 (95% confidence interval 1.04 to 1.14; I²=68%; τ²=0.021; 95% prediction interval 0.81 to 1.46). The prediction interval indicated that the difference could be much more substantial, in either direction. We further explored heterogeneity and found that PI/ECO dissimilarities, especially for the comparisons of dietary supplements in randomised controlled trials and nutrient status in cohort studies, explained most of the differences. When the type of intake or exposure between both types of evidence was identical, the estimates were similar. For continuous outcomes, small differences were observed between randomised controlled trials and cohort studies.

Conclusion

On average, the difference in pooled results between estimates from BoE(RCT) and BoE(CS) was small. But wide prediction intervals and some substantial statistical heterogeneity in cohort studies indicate that important differences or potential bias in individual comparisons or studies cannot be excluded. Observed differences were mainly driven by dissimilarities in population, intervention or exposure, comparator, and outcome. These findings could help researchers further understand the integration of such evidence into prospective nutrition evidence syntheses and improve evidence based dietary guidelines.

SUBMITTER: Schwingshackl L

PROVIDER: S-EPMC8441535 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundRandomized controlled trials (RCTs) and cohort studies are the most common study design types used to assess the treatment effects of medical interventions. To evaluate the agreement of effect estimates between bodies of evidence (BoE) from randomized controlled trials (RCTs) and cohort studies and to identify factors associated with disagreement.MethodsSystematic reviews were published in the 13 medical journals with the highest impact factor identified through a MEDLINE search. BoE-pairs from RCTs and cohort studies with the same medical research question were included. We rated the similarity of PI/ECO (Population, Intervention/Exposure, Comparison, Outcome) between BoE from RCTs and cohort studies. The agreement of effect estimates across BoE was analyzed by pooling ratio of ratios (RoR) for binary outcomes and difference of mean differences for continuous outcomes. We performed subgroup analyses to explore factors associated with disagreements.ResultsOne hundred twenty-nine BoE pairs from 64 systematic reviews were included. PI/ECO-similarity degree was moderate: two BoE pairs were rated as "more or less identical"; 90 were rated as "similar but not identical" and 37 as only "broadly similar". For binary outcomes, the pooled RoR was 1.04 (95% CI 0.97-1.11) with considerable statistical heterogeneity. For continuous outcomes, differences were small. In subgroup analyses, degree of PI/ECO-similarity, type of intervention, and type of outcome, the pooled RoR indicated that on average, differences between both BoE were small. Subgroup analysis by degree of PI/ECO-similarity revealed high statistical heterogeneity and wide prediction intervals across PI/ECO-dissimilar BoE pairs.ConclusionsOn average, the pooled effect estimates between RCTs and cohort studies did not differ. Statistical heterogeneity and wide prediction intervals were mainly driven by PI/ECO-dissimilarities (i.e., clinical heterogeneity) and cohort studies. The potential influence of risk of bias and certainty of the evidence on differences of effect estimates between RCTs and cohort studies needs to be explored in upcoming meta-epidemiological studies.

Project description:We aimed to identify and compare empirical data to determine the concordance of diet-disease effect estimates of bodies of evidence (BoE) from randomized controlled trials (RCTs), dietary intake, and biomarkers of dietary intake in cohort studies (CSs). The Cochrane Database of Systematic Reviews and MEDLINE were searched for systematic reviews (SRs) of RCTs and SRs of CSs that investigated both dietary intake and biomarkers of intake published between 1 January 2010 and 31 December 2019. For matched diet-disease associations, the concordance between results from the 3 different BoE was analyzed using 2 definitions: qualitative (e.g., 95% CI within a predefined range) and quantitative (test hypothesis on the z score). Moreover, the differences in the results coming from BoERCTs, BoECSs dietary intake, and BoECSs biomarkers were synthesized to get a pooled ratio of risk ratio (RRR) across all eligible diet-disease associations, so as to compare the 3 BoE. Overall, 49 diet-disease associations derived from 41 SRs were identified and included in the analysis. Twenty-four percent, 10%, and 39% of the diet-disease associations were qualitatively concordant comparing BoERCTs with BoECSs dietary intake, BoERCTs with BoECSs biomarkers, and comparing both BoE from CSs, respectively; 88%, 69%, and 90% of the diet-disease associations were quantitatively concordant comparing BoERCTs with BoECSs dietary intake, BoERCTs with BoECSs biomarkers, and comparing both BoE from CSs, respectively. The pooled RRRs comparing effects from BoERCTs with effects from BoECSs dietary intake were 1.09 (95% CI: 1.06, 1.13) and 1.18 (95% CI: 1.10, 1.25) compared with BoECSs biomarkers. Comparing both BoE from CSs, the difference in the results was also small (RRR: 0.92; 95% CI: 0.88, 0.96). Our findings suggest that BoE from RCTs and CSs are often quantitatively concordant. Prospective SRs in nutrition research should include, whenever possible, BoE from RCTs and CSs on dietary intake and biomarkers of intake to provide the whole picture for an investigated diet-disease association.

Project description:IntroductionAlthough interest in including non-randomised studies of interventions (NRSIs) in meta-analysis of randomised controlled trials (RCTs) is growing, estimates of effectiveness obtained from NRSIs are vulnerable to greater bias than RCTs. The objectives of this study are to: (1) explore how NRSIs can be integrated into a meta-analysis of RCTs; (2) assess concordance of the evidence from non-randomised and randomised trials and explore factors associated with agreement; and (3) investigate the impact on estimates of pooled bodies of evidence when NRSIs are included.Methods and analysisWe will conduct a systematic survey of 210 systematic reviews that include both RCTs and NRSIs, published from 2017 to 2022. We will randomly select reviews, stratified in a 1:1 ratio by Core vs non-Core clinical journals, as defined by the National Library of Medicine. Teams of paired reviewers will independently determine eligibility and abstract data using standardised, pilot-tested forms. The concordance of the evidence will be assessed by exploring agreement in the relative effect reported by NRSIs and RCT addressing the same clinical question, defined as similarity of the population, intervention/exposure, control and outcomes. We will conduct univariable and multivariable logistic regression analyses to examine the association of prespecified study characteristics with agreement in the estimates between NRSIs and RCTs. We will calculate the ratio of the relative effect estimate from NRSIs over that from RCTs, along with the corresponding 95% CI. We will use a bias-corrected meta-analysis model to investigate the influence on pooled estimates when NRSIs are included in the evidence synthesis.Ethics and disseminationEthics approval is not required. The findings of this study will be disseminated through peer-reviewed publications, conference presentations and condensed summaries for clinicians, health policymakers and guideline developers regarding the design, conduct, analysis, and interpretation of meta-analysis that integrate RCTs and NRSIs.

Project description:Randomized controlled trials (RCTs) and cohort studies are the most common study design types used to assess treatment effects of medical interventions. We aimed to hypothetically pool bodies of evidence (BoE) from RCTs with matched BoE from cohort studies included in the same systematic review. BoE derived from systematic reviews of RCTs and cohort studies published in the 13 medical journals with the highest impact factor were considered. We re-analyzed effect estimates of the included systematic reviews by pooling BoE from RCTs with BoE from cohort studies using random and common effects models. We evaluated statistical heterogeneity, 95% prediction intervals, weight of BoE from RCTs to the pooled estimate, and whether integration of BoE from cohort studies modified the conclusion from BoE of RCTs. Overall, 118 BoE-pairs based on 653 RCTs and 804 cohort studies were pooled. By pooling BoE from RCTs and cohort studies with a random effects model, for 61 (51.7%) out of 118 BoE-pairs, the 95% confidence interval (CI) excludes no effect. By pooling BoE from RCTs and cohort studies, the median I2 was 48%, and the median contributed percentage weight of RCTs to the pooled estimates was 40%. The direction of effect between BoE from RCTs and pooled effect estimates was mainly concordant (79.7%). The integration of BoE from cohort studies modified the conclusion (by examining the 95% CI) from BoE of RCTs in 32 (27%) of the 118 BoE-pairs, but the direction of effect was mainly concordant (88%). Our findings provide insights for the potential impact of pooling both BoE in systematic reviews. In medical research, it is often important to rely on both evidence of RCTs and cohort studies to get a whole picture of an investigated intervention-disease association. A decision for or against pooling different study designs should also always take into account, for example, PI/ECO similarity, risk of bias, coherence of effect estimates, and also the trustworthiness of the evidence. Overall, there is a need for more research on the influence of those issues on potential pooling.

Project description:ObjectiveTo synthesise evidence on the average bias and heterogeneity associated with reported methodological features of randomized trials.DesignSystematic review of meta-epidemiological studies.MethodsWe retrieved eligible studies included in a recent AHRQ-EPC review on this topic (latest search September 2012), and searched Ovid MEDLINE and Ovid EMBASE for studies indexed from Jan 2012-May 2015. Data were extracted by one author and verified by another. We combined estimates of average bias (e.g. ratio of odds ratios (ROR) or difference in standardised mean differences (dSMD)) in meta-analyses using the random-effects model. Analyses were stratified by type of outcome ("mortality" versus "other objective" versus "subjective"). Direction of effect was standardised so that ROR < 1 and dSMD < 0 denotes a larger intervention effect estimate in trials with an inadequate or unclear (versus adequate) characteristic.ResultsWe included 24 studies. The available evidence suggests that intervention effect estimates may be exaggerated in trials with inadequate/unclear (versus adequate) sequence generation (ROR 0.93, 95% CI 0.86 to 0.99; 7 studies) and allocation concealment (ROR 0.90, 95% CI 0.84 to 0.97; 7 studies). For these characteristics, the average bias appeared to be larger in trials of subjective outcomes compared with other objective outcomes. Also, intervention effects for subjective outcomes appear to be exaggerated in trials with lack of/unclear blinding of participants (versus blinding) (dSMD -0.37, 95% CI -0.77 to 0.04; 2 studies), lack of/unclear blinding of outcome assessors (ROR 0.64, 95% CI 0.43 to 0.96; 1 study) and lack of/unclear double blinding (ROR 0.77, 95% CI 0.61 to 0.93; 1 study). The influence of other characteristics (e.g. unblinded trial personnel, attrition) is unclear.ConclusionsCertain characteristics of randomized trials may exaggerate intervention effect estimates. The average bias appears to be greatest in trials of subjective outcomes. More research on several characteristics, particularly attrition and selective reporting, is needed.

Project description:There is a great deal of interest in personalized, individualized, or precision interventions for disease and health-risk mitigation. This is as true of nutrition-based intervention and prevention strategies as it is for pharmacotherapies and pharmaceutical-oriented prevention strategies. Essentially, technological breakthroughs have enabled researchers to probe an individual's unique genetic, biochemical, physiological, behavioral, and exposure profile, allowing them to identify very specific and often nuanced factors that an individual might possess, which may make it more or less likely that he or she responds favorably to a particular intervention (e.g., nutrient supplementation) or disease prevention strategy (e.g., specific diet). However, as compelling and intuitive as personalized nutrition might be in the current era in which data-intensive biomedical characterization of individuals is possible, appropriately and objectively vetting personalized nutrition strategies is not trivial and requires novel study designs and data analytical methods. These designs and methods must consider a very integrated use of the multiple contemporary biomedical assays and technologies that motivate them, which adds to their complexity. Single-subject or N-of-1 trials can be used to assess the utility of personalized interventions and, in addition, can be crafted in such a way as to accommodate the necessarily integrated use of many emerging biomedical technologies and assays. In this review, we consider the motivation, design, and implementation of N-of-1 trials in translational nutrition research that are meant to assess the utility of personalized nutritional strategies. We provide a number of example studies, discuss appropriate analytical methods given the complex data they generate and require, and consider how such studies could leverage integration of various biomarker assays and clinical end points. Importantly, we also consider the development of strategies and algorithms for matching nutritional needs to individual biomedical profiles and the issues surrounding them. Finally, we discuss the limitations of personalized nutrition studies, possible extensions of N-of-1 nutritional intervention studies, and areas of future research.

Project description:BackgroundRecruitment of patients is one of the main challenges when designing and conducting randomised controlled trials (RCTs). Trials of rare injuries or those that include surgical interventions pose added challenges due to the small number of potentially eligible patients and issues with patient preferences and surgeon equipoise. We explore key issues to consider when recruiting to orthopaedic surgical trials from the perspective of staff and patients with the aim of informing the development of strategies to improve recruitment in future research.DesignTwo qualitative process evaluations of a UK-wide orthopaedic surgical RCT (ACTIVE) and mixed methods randomised feasibility study (PRESTO). Qualitative semi-structured interviews were conducted and data was analysed thematically.SettingNHS secondary care organisations throughout the UK. Interviews were undertaken via telephone.ParticipantsThirty-seven health professionals including UK-based spinal and orthopaedic surgeons and individuals involved in recruitment to the ACTIVE and PRESTO studies (e.g. research nurses, surgeons, physiotherapists). Twenty-two patients including patients who agreed to participate in the ACTIVE and PRESTO studies (n=15) and patients that declined participation in the ACTIVE study (n=7) were interviewed.ResultsWe used a mixed methods systematic review of recruiting patients to randomised controlled trials as a framework for reporting and analysing our findings. Our findings mapped onto those identified in the systematic review and highlighted the importance of equipoise, randomisation, communication, patient's circumstances, altruism and trust in clinical and research teams. Our findings also emphasised the importance of considering how eligibility criteria are operationalised and the impact of complex patient pathways when recruiting to surgical trials. In particular, the influence of health professionals, who are not involved in trial recruitment, on patients' treatment preferences by suggesting they would receive a certain treatment ahead of recruitment consultations should not be underestimated.ConclusionsA wealth of evidence exploring factors affecting recruitment to randomised controlled trials exists. A methodological shift is now required to ensure that this evidence is used by all those involved in recruitment and to ensure that existing knowledge is translated into methods for optimising recruitment to future trials.Trial registrationACTIVE: ( ISRCTN98152560 ). Registered on 06/03/2018. PRESTO: ( ISRCTN12094890 ). Registered on 22/02/2018.

Project description:ObjectivesTo determine whether an association exists between the number of published randomised controlled trials and the global burden of disease, whether certain diseases are under-investigated relative to their burden, and whether the relation between the output of randomised trials and global burden of disease can be explained by the relative disease burden in high and low income regions.DesignCross sectional investigation.Study sampleAll primary reports of randomised trials published in December 2012 and indexed in PubMed by 17 November 2013.Main outcome measuresNumber of trials conducted and number of participants randomised for each of 239 different diseases or injuries; variation in each outcome explainable by total disability adjusted life years (a measure of the overall burden of each disease) and the ratio of disability adjusted life years in low income to high income regions (a measure of whether a disease is more likely to affect people living in high income regions) quantified using multivariable regression.Results4190 abstracts were reviewed and 1351 primary randomised trials identified, of which 1097 could be classified using the global burden of disease taxonomy. Total disability adjusted life years was poorly associated with number of randomised trials and number of participants randomised in univariable analysis (Spearman's r=0.35 and 0.33, respectively), although it was a significant predictor in the univariable and multivariable models (P<0.001). Diseases for which the burden was predominantly located in low income regions had sevenfold fewer trials per million disability adjusted life years than diseases predominantly located in high income regions. However, only 26% of the variation in number of trials among diseases could be explained by total disability adjusted life years and the ratio of disability adjusted life years in low income regions to high income regions. Many high income type diseases (for example, neck pain, glomerulonephritis) have proportionally fewer randomised trials compared with low income type diseases (for example, vitamin A deficiency).ConclusionsOverall, a weak association existed between global burden of disease and number of published randomised trials. A global observatory for research is needed to monitor and reduce the discordance between the output of randomised trials and global burden of disease.

Project description:BackgroundThe Cochrane risk of bias (CROB) tool and Physiotherapy Evidence Database (PEDro) scale are used to evaluate risk of bias of randomized controlled trials. We assessed the level of agreement between the instruments.MethodsWe searched the Cochrane Library to identify trials included in systematic reviews evaluating physical therapy interventions. For trials that met our inclusion criteria (primary reference in Cochrane review, review used CROB (2008 version), indexed in PEDro), CROB items were extracted from the reviews and PEDro items and total score were downloaded from PEDro. Kappa statistics were used to determine the agreement between CROB and PEDro scale items that evaluate similar constructs (e.g., randomization). The total PEDro score was compared to the CROB summary score (% of items met) using an Intraclass Correlation Coefficient. Sensitivity analyses explored the impact of the CROB "unclear" category and variants of CROB blinding items. Kappa statistics were used to determine agreement between different thresholds for "acceptable" risk of bias between CROB and PEDro scale summary scores.ResultsWe included 1442 trials from 108 Cochrane reviews. Agreement was "moderate" for three of the six CROB and PEDro scale items that evaluate similar constructs (allocation concealment, participant blinding, assessor blinding; Kappa = 0.479-0.582). Agreement between the summary scores was "poor" (Intraclass Correlation Coefficient = 0.285). Agreement was highest when the CROB "unclear" category was collapsed with "high" and when participant, personnel and assessor blinding were evaluated separately in CROB. Agreement for different thresholds for "acceptable" risk of bias between CROB and PEDro summary scores was, at best, "fair".ConclusionThere was moderate agreement for half of the PEDro and CROB items that evaluate similar constructs. Interpretation of the CROB "unclear" category and variants of the CROB blinding items substantially influenced agreement. Either instrument can be used to quantify risk of bias, but they can't be used interchangeably.