Dataset Information

How to design a pre-specified statistical analysis approach to limit p-hacking in clinical trials: the Pre-SPEC framework.

ABSTRACT: Results from clinical trials can be susceptible to bias if investigators choose their analysis approach after seeing trial data, as this can allow them to perform multiple analyses and then choose the method that provides the most favourable result (commonly referred to as 'p-hacking'). Pre-specification of the planned analysis approach is essential to help reduce such bias, as it ensures analytical methods are chosen in advance of seeing the trial data. For this reason, guidelines such as SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) and ICH-E9 (International Conference for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use) require the statistical methods for a trial's primary outcome be pre-specified in the trial protocol. However, pre-specification is only effective if done in a way that does not allow p-hacking. For example, investigators may pre-specify a certain statistical method such as multiple imputation, but give little detail on how it will be implemented. Because there are many different ways to perform multiple imputation, this approach to pre-specification is ineffective, as it still allows investigators to analyse the data in different ways before deciding on a final approach. In this article, we describe a five-point framework (the Pre-SPEC framework) for designing a pre-specified analysis approach that does not allow p-hacking. This framework was designed based on the principles in the SPIRIT and ICH-E9 guidelines and is intended to be used in conjunction with these guidelines to help investigators design the statistical analysis strategy for the trial's primary outcome in the trial protocol.

SUBMITTER: Kahan BC

PROVIDER: S-EPMC7487509 | biostudies-literature | 2020 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

How to design a pre-specified statistical analysis approach to limit p-hacking in clinical trials: the Pre-SPEC framework.

Kahan Brennan C BC Forbes Gordon G Cro Suzie S

BMC medicine 20200907 1

Results from clinical trials can be susceptible to bias if investigators choose their analysis approach after seeing trial data, as this can allow them to perform multiple analyses and then choose the method that provides the most favourable result (commonly referred to as 'p-hacking'). Pre-specification of the planned analysis approach is essential to help reduce such bias, as it ensures analytical methods are chosen in advance of seeing the trial data. For this reason, guidelines such as SPIRI ...[more]

PMID: 32892743

Similar Datasets

Project description:BACKGROUND:In proportionate or adaptive interventions, the dose or intensity can be adjusted based on individual need at predefined decision stages during the delivery of the intervention. The development of such interventions may require an evaluation of the effectiveness of the individual stages in addition to the whole intervention. However, evaluating individual stages of an intervention has various challenges, particularly the statistical design and analysis. This review aimed to identify the use of trials of proportionate interventions and how they are being designed and analysed in current practice. METHODS:We searched MEDLINE, Web of Science and PsycINFO for articles published between 2010 and 2015 inclusive. We considered trials of proportionate interventions in all fields of research. For each trial, its aims, design and analysis were extracted. The data synthesis was conducted using summary statistics and a narrative format. RESULTS:Our review identified 44 proportionate intervention trials, comprising 28 trial results, 13 protocols and three secondary analyses. These were mostly described as stepped care (n=37) and mainly focussed on mental health research (n=30). The other studies were aimed at finding an optimal adaptive treatment strategy (n=7) in a variety of therapeutic areas. Further terminology used included adaptive intervention, staged intervention, sequentially multiple assignment trial or a two-phase design. The median number of decision stages in the interventions was two and only one study explicitly evaluated the effect of the individual stages. CONCLUSIONS:Trials of proportionate staged interventions are being used predominantly within the mental health field. However, few studies consider the different stages of the interventions, either at the design or the analysis phase, and how they may interact with one another. There is a need for further guidance on the design, analyses and reporting across trials of proportionate interventions. TRIAL REGISTRATION:Prospero, CRD42016033781. Registered on 2 February 2016.

Project description:PurposeTo develop a statistical approach that compares patient selection strategies across clinical trials and apply this approach to acute ischemic stroke clinical trials to identify the optimal inclusion criteria.MethodsWe developed a statistical approach that compares the number needed to treat to achieve one success (NNT) along with the number needed to screen to achieve one success (NNS) and assesses if there are significant differences in inclusion criteria, treatment course, and clinical outcome among patients that may have been included/excluded in the trials. We applied this approach to the study population from four recent positive acute stroke clinical trials: MR CLEAN, EXTEND-IA, ESCAPE, and SWIFT PRIME, applying published trial criteria to an independent registry of 612 acute stroke patients, since we did not have access to the complete trial data.ResultsAlthough reported NNT were similar for EXTEND-IA, SWIFT PRIME and ESCAPE, and somewhat higher for MR CLEAN, NNS varied across the trials from 21 for EXTEND-IA, 27 for MR CLEAN, to 46 for ESCAPE and 64 for SWIFT PRIME, reflecting less and more stringent inclusion criteria, respectively. Although there were significant differences in imaging biomarkers and other clinical characteristics among patients that may have been included/excluded in the trials, these differences did not translate to significant differences in treatment course or clinical outcomes.ConclusionsOur study proposes a robust statistical approach that can be applied to a larger pooled trial dataset, if made available, to objectively compare across clinical trials and inform inclusion criteria of future trials. Pooled analysis of the acute stroke trial data is needed to determine which imaging biomarker inclusion criteria are critical and which may be relaxed. If this procedure were applied across the pooled trial data, it could decrease costs and refine the design of future trials to be the most efficacious for the greatest number of patients.

Dataset Information

How to design a pre-specified statistical analysis approach to limit p-hacking in clinical trials: the Pre-SPEC framework.

Publications

How to design a pre-specified statistical analysis approach to limit p-hacking in clinical trials: the Pre-SPEC framework.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets