Dataset Information

Evidence of unexplained discrepancies between planned and conducted statistical analyses: a review of randomised trials.

ABSTRACT: BACKGROUND:Choosing or altering the planned statistical analysis approach after examination of trial data (often referred to as 'p-hacking') can bias the results of randomised trials. However, the extent of this issue in practice is currently unclear. We conducted a review of published randomised trials to evaluate how often a pre-specified analysis approach is publicly available, and how often the planned analysis is changed. METHODS:A review of randomised trials published between January and April 2018 in six leading general medical journals. For each trial, we established whether a pre-specified analysis approach was publicly available in a protocol or statistical analysis plan and compared this to the trial publication. RESULTS:Overall, 89 of 101 eligible trials (88%) had a publicly available pre-specified analysis approach. Only 22/89 trials (25%) had no unexplained discrepancies between the pre-specified and conducted analysis. Fifty-four trials (61%) had one or more unexplained discrepancies, and in 13 trials (15%), it was impossible to ascertain whether any unexplained discrepancies occurred due to incomplete reporting of the statistical methods. Unexplained discrepancies were most common for the analysis model (n?=?31, 35%) and analysis population (n?=?28, 31%), followed by the use of covariates (n?=?23, 26%) and the approach for handling missing data (n?=?16, 18%). Many protocols or statistical analysis plans were dated after the trial had begun, so earlier discrepancies may have been missed. CONCLUSIONS:Unexplained discrepancies in the statistical methods of randomised trials are common. Increased transparency is required for proper evaluation of results.

SUBMITTER: Cro S

PROVIDER: S-EPMC7257229 | biostudies-literature | 2020 May

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Evidence of unexplained discrepancies between planned and conducted statistical analyses: a review of randomised trials.

Cro Suzie S Forbes Gordon G Johnson Nicholas A NA Kahan Brennan C BC

BMC medicine 20200529 1

<h4>Background</h4>Choosing or altering the planned statistical analysis approach after examination of trial data (often referred to as 'p-hacking') can bias the results of randomised trials. However, the extent of this issue in practice is currently unclear. We conducted a review of published randomised trials to evaluate how often a pre-specified analysis approach is publicly available, and how often the planned analysis is changed.<h4>Methods</h4>A review of randomised trials published betwee ...[more]

PMID: 32466758

Similar Datasets

Project description:IntroductionCohort studies generate and collect longitudinal data for a variety of research purposes. Randomised controlled trials (RCTs) increasingly use cohort studies as data infrastructures to help identify and recruit trial participants and assess outcomes.ObjectiveTo examine the extent, range and nature of research using cohorts for RCTs and describe the varied definitions and conceptual boundaries for RCTs using cohorts.DesignScoping review.Data sourcesSearches were undertaken in January 2021 in MEDLINE (Ovid) and EBM Reviews-Cochrane Methodology Registry (Final issue, third Quarter 2012).Eligibility criteriaReports published between January 2007 and December 2021 of (a) cohorts used or planned to be used, to conduct RCTs, or (b) RCTs which use cohorts to recruit participants and/or collect trial outcomes, or (c) methodological studies discussing the use of cohorts for RCTs.Data extraction and synthesisData were extracted on the condition being studied, age group, setting, country/continent, intervention(s) and comparators planned or received, unit of randomisation, timing of randomisation, approach to informed consent, study design and terminology.ResultsA total of 175 full-text articles were assessed for eligibility. We identified 61 protocols, 9 descriptions of stand-alone cohorts intended to be used for future RCTs, 39 RCTs using cohorts and 34 methodological papers.The use and scope of this approach is growing. The thematics of study are far-ranging, including population health, oncology, mental and behavioural disorders, and musculoskeletal conditions.Authors reported that this approach can lead to more efficient recruitment, more representative samples, and lessen disappointment bias and crossovers.ConclusionThis review outlines the development of cohorts to conduct RCTs including the range of use and innovative changes and adaptations. Inconsistencies in the use of terminology and concepts are highlighted. Guidance now needs to be developed to support the design and reporting of RCTs conducted using cohorts.

Project description:IntroductionRandomised controlled trials (RCTs) aim to assess the effect of one (or more) unproven health interventions relative to other reference interventions. RCTs sometimes use an ordinal outcome, which is an endpoint that comprises of multiple, monotonically ordered categories that are not necessarily separated by a quantifiable distance. Ordinal outcomes are appealing in clinical settings as specific disease states can represent meaningful categories that may be of clinical importance to researchers. Ordinal outcomes can also retain information and increase statistical power compared to dichotomised outcomes and can allow multiple clinical outcomes to be comprised in a single endpoint. Target parameters for ordinal outcomes in RCTs may vary depending on the nature of the research question, the modelling assumptions and the expertise of the data analyst. The aim of this scoping review is to systematically describe the use of ordinal outcomes in contemporary RCTs. Specifically, we aim to: [Formula: see text] Identify which target parameters are of interest in trials that use an ordinal outcome, and whether these parameters are explicitly defined. [Formula: see text] Describe how ordinal outcomes are analysed in RCTs to estimate a treatment effect. [Formula: see text] Describe whether RCTs that use an ordinal outcome adequately report key methodological aspects specific to the analysis of the ordinal outcome. Results from this review will outline the current state of practice of the use of ordinal outcomes in RCTs. Ways to improve the analysis and reporting of ordinal outcomes in RCTs will be discussed.Methods and analysisWe will review RCTs that are published in the top four medical journals (British Medical Journal, New England Journal of Medicine, The Lancet and the Journal of the American Medical Association) between 1 January 2012 and 31 July 2022 that use an ordinal outcome as either a primary or a secondary outcome. The review will identify articles through a PubMed-specific search strategy. Our review will adhere to guidelines for scoping reviews as described in the PRISMA-ScR checklist. The study characteristics and details of the study design and analysis, including the target parameter(s) and statistical methods used to analyse the ordinal outcome, will be extracted from eligible studies. The screening, review and data extraction will be conducted using Covidence, a web-based tool for managing systematic reviews. The data will be summarised using descriptive statistics.

Project description:BackgroundMost publications about selective reporting in clinical trials have focussed on outcomes. However, selective reporting of analyses for a given outcome may also affect the validity of findings. If analyses are selected on the basis of the results, reporting bias may occur. The aims of this study were to review and summarise the evidence from empirical cohort studies that assessed discrepant or selective reporting of analyses in randomised controlled trials (RCTs).Methods and findingsA systematic review was conducted and included cohort studies that assessed any aspect of the reporting of analyses of RCTs by comparing different trial documents, e.g., protocol compared to trial report, or different sections within a trial publication. The Cochrane Methodology Register, Medline (Ovid), PsycInfo (Ovid), and PubMed were searched on 5 February 2014. Two authors independently selected studies, performed data extraction, and assessed the methodological quality of the eligible studies. Twenty-two studies (containing 3,140 RCTs) published between 2000 and 2013 were included. Twenty-two studies reported on discrepancies between information given in different sources. Discrepancies were found in statistical analyses (eight studies), composite outcomes (one study), the handling of missing data (three studies), unadjusted versus adjusted analyses (three studies), handling of continuous data (three studies), and subgroup analyses (12 studies). Discrepancy rates varied, ranging from 7% (3/42) to 88% (7/8) in statistical analyses, 46% (36/79) to 82% (23/28) in adjusted versus unadjusted analyses, and 61% (11/18) to 100% (25/25) in subgroup analyses. This review is limited in that none of the included studies investigated the evidence for bias resulting from selective reporting of analyses. It was not possible to combine studies to provide overall summary estimates, and so the results of studies are discussed narratively.ConclusionsDiscrepancies in analyses between publications and other study documentation were common, but reasons for these discrepancies were not discussed in the trial reports. To ensure transparency, protocols and statistical analysis plans need to be published, and investigators should adhere to these or explain discrepancies.

Project description:BACKGROUND:Selective reporting of outcomes in clinical trials is a serious problem. We aimed to investigate the influence of the peer review process within biomedical journals on reporting of primary outcome(s) and statistical analyses within reports of randomised trials. METHODS:Each month, PubMed (May 2014 to April 2015) was searched to identify primary reports of randomised trials published in six high-impact general and 12 high-impact specialty journals. The corresponding author of each trial was invited to complete an online survey asking authors about changes made to their manuscript as part of the peer review process. Our main outcomes were to assess: (1) the nature and extent of changes as part of the peer review process, in relation to reporting of the primary outcome(s) and/or primary statistical analysis; (2) how often authors followed these requests; and (3) whether this was related to specific journal or trial characteristics. RESULTS:Of 893 corresponding authors who were invited to take part in the online survey 258 (29%) responded. The majority of trials were multicentre (n?=?191; 74%); median sample size 325 (IQR 138 to 1010). The primary outcome was clearly defined in 92% (n?=?238), of which the direction of treatment effect was statistically significant in 49%. The majority responded (1-10 Likert scale) they were satisfied with the overall handling (mean 8.6, SD 1.5) and quality of peer review (mean 8.5, SD 1.5) of their manuscript. Only 3% (n?=?8) said that the editor or peer reviewers had asked them to change or clarify the trial's primary outcome. However, 27% (n?=?69) reported they were asked to change or clarify the statistical analysis of the primary outcome; most had fulfilled the request, the main motivation being to improve the statistical methods (n?=?38; 55%) or avoid rejection (n?=?30; 44%). Overall, there was little association between authors being asked to make this change and the type of journal, intervention, significance of the primary outcome, or funding source. Thirty-six percent (n?=?94) of authors had been asked to include additional analyses that had not been included in the original manuscript; in 77% (n?=?72) these were not pre-specified in the protocol. Twenty-three percent (n?=?60) had been asked to modify their overall conclusion, usually (n?=?53; 88%) to provide a more cautious conclusion. CONCLUSION:Overall, most changes, as a result of the peer review process, resulted in improvements to the published manuscript; there was little evidence of a negative impact in terms of post hoc changes of the primary outcome. However, some suggested changes might be considered inappropriate, such as unplanned additional analyses, and should be discouraged.

Project description:OBJECTIVE:Randomised controlled trials (RCT) are the gold standard to provide unbiased data. However, when patients have a treatment preference, randomisation may influence participation and outcomes (eg, external and internal validity). The aim of this study was to assess the influence of patients' preference in RCTs by analysing partially randomised patient preference trials (RPPT); an RCT and preference cohort combined. DESIGN:Systematic review and meta-analyses. DATA SOURCES:MEDLINE, Embase, PsycINFO and the Cochrane Library. ELIGIBILITY CRITERIA FOR SELECTING STUDIES:RPPTs published between January 2005 and October 2018 reporting on allocation of patients to randomised and preference cohorts were included. DATA EXTRACTION AND SYNTHESIS:Two independent reviewers extracted data. The main outcomes were the difference in external validity (participation and baseline characteristics) and internal validity (lost to follow-up, crossover and the primary outcome) between the randomised and the preference cohort within each RPPT, compared in a meta-regression using a Wald test. Risk of bias was not assessed, as no quality assessment for RPPTs has yet been developed. RESULTS:In total, 117 of 3734 identified articles met screening criteria and 44 were eligible (24 873 patients). The participation rate in RPPTs was >95% in 14 trials (range: 48%-100%) and the randomisation refusal rate was >50% in 26 trials (range: 19%-99%). Higher education, female, older age, race and prior experience with one treatment arm were characteristics of patients declining randomisation. The lost to follow-up and cross-over rate were significantly higher in the randomised cohort compared with the preference cohort. Following the meta-analysis, the reported primary outcomes were comparable between both cohorts of the RPPTs, mean difference 0.093 (95% CI -0.178 to 0.364, p=0.502). CONCLUSIONS:Patients' preference led to a substantial proportion of a specific patient group refusing randomisation, while it did not influence the primary outcome within an RPPT. Therefore, RPPTs could increase external validity without compromising the internal validity compared with RCTs. PROSPERO REGISTRATION NUMBER:CRD42019094438.

Dataset Information

Evidence of unexplained discrepancies between planned and conducted statistical analyses: a review of randomised trials.

Publications

Evidence of unexplained discrepancies between planned and conducted statistical analyses: a review of randomised trials.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets