Dataset Information

How to design a pre-specified statistical analysis approach to limit p-hacking in clinical trials: the Pre-SPEC framework.

ABSTRACT: Results from clinical trials can be susceptible to bias if investigators choose their analysis approach after seeing trial data, as this can allow them to perform multiple analyses and then choose the method that provides the most favourable result (commonly referred to as 'p-hacking'). Pre-specification of the planned analysis approach is essential to help reduce such bias, as it ensures analytical methods are chosen in advance of seeing the trial data. For this reason, guidelines such as SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) and ICH-E9 (International Conference for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use) require the statistical methods for a trial's primary outcome be pre-specified in the trial protocol. However, pre-specification is only effective if done in a way that does not allow p-hacking. For example, investigators may pre-specify a certain statistical method such as multiple imputation, but give little detail on how it will be implemented. Because there are many different ways to perform multiple imputation, this approach to pre-specification is ineffective, as it still allows investigators to analyse the data in different ways before deciding on a final approach. In this article, we describe a five-point framework (the Pre-SPEC framework) for designing a pre-specified analysis approach that does not allow p-hacking. This framework was designed based on the principles in the SPIRIT and ICH-E9 guidelines and is intended to be used in conjunction with these guidelines to help investigators design the statistical analysis strategy for the trial's primary outcome in the trial protocol.

SUBMITTER: Kahan BC

PROVIDER: S-EPMC7487509 | biostudies-literature | 2020 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

How to design a pre-specified statistical analysis approach to limit p-hacking in clinical trials: the Pre-SPEC framework.

Kahan Brennan C BC Forbes Gordon G Cro Suzie S

BMC medicine 20200907 1

Results from clinical trials can be susceptible to bias if investigators choose their analysis approach after seeing trial data, as this can allow them to perform multiple analyses and then choose the method that provides the most favourable result (commonly referred to as 'p-hacking'). Pre-specification of the planned analysis approach is essential to help reduce such bias, as it ensures analytical methods are chosen in advance of seeing the trial data. For this reason, guidelines such as SPIRI ...[more]

PMID: 32892743

Similar Datasets

Project description:BackgroundData harmonization is a powerful method to equilibrate items in measures that evaluate the same underlying construct. There are multiple measures to evaluate dementia related behavioral symptoms. Pre-statistical harmonization of behavioral instruments in dementia research is the first step to develop a statistical crosswalk between measures. Studies that conduct pre-statistical harmonization of behavioral instruments rarely document their methods in a structured, reproducible manner. This is a crucial step which entails careful review, documentation and scrutiny of source data to ensure sufficient comparability between items prior to data pooling. Here, we document the pre-statistical harmonization of items measuring behavioral and psychological symptoms among people with dementia. We provide a box of recommended procedure for future studies.MethodsWe identified behavioral instruments that are used in clinical practice, a national survey, and randomized trials of dementia care interventions. We rigorously reviewed question content and scoring procedures to establish sufficient comparability across items as well as item quality prior to data pooling. Additionally, we standardized coding to Stata-readable format, which allowed us to automate approaches to identify potential cross-study differences in items and low-quality items. To ensure reasonable model fit for statistical co-calibration, we estimated two-parameter logistic Item Response Theory models within each of the eight studies.ResultsWe identified 59 items from 11 behavioral instruments across the eight datasets. We found considerable cross-study heterogeneity in administration and coding procedures for items that measure the same attribute. Discrepancies existed in terms of directionality and quantification of behavioral symptoms for even seemingly comparable items. We resolved item response heterogeneity, missingness and skewness, conditional dependency prior to estimation of item response theory models for statistical co-calibration. We used several rigorous data transformation procedures to address these issues, including re-coding and truncation.ConclusionsThis study highlights the importance of each aspect involved in the pre-statistical harmonization process of behavioral instruments. We provide guidelines and recommendations for how future research may detect and account for similar issues in pooling behavioral and related instruments.

Project description:BackgroundIn proportionate or adaptive interventions, the dose or intensity can be adjusted based on individual need at predefined decision stages during the delivery of the intervention. The development of such interventions may require an evaluation of the effectiveness of the individual stages in addition to the whole intervention. However, evaluating individual stages of an intervention has various challenges, particularly the statistical design and analysis. This review aimed to identify the use of trials of proportionate interventions and how they are being designed and analysed in current practice.MethodsWe searched MEDLINE, Web of Science and PsycINFO for articles published between 2010 and 2015 inclusive. We considered trials of proportionate interventions in all fields of research. For each trial, its aims, design and analysis were extracted. The data synthesis was conducted using summary statistics and a narrative format.ResultsOur review identified 44 proportionate intervention trials, comprising 28 trial results, 13 protocols and three secondary analyses. These were mostly described as stepped care (n=37) and mainly focussed on mental health research (n=30). The other studies were aimed at finding an optimal adaptive treatment strategy (n=7) in a variety of therapeutic areas. Further terminology used included adaptive intervention, staged intervention, sequentially multiple assignment trial or a two-phase design. The median number of decision stages in the interventions was two and only one study explicitly evaluated the effect of the individual stages.ConclusionsTrials of proportionate staged interventions are being used predominantly within the mental health field. However, few studies consider the different stages of the interventions, either at the design or the analysis phase, and how they may interact with one another. There is a need for further guidance on the design, analyses and reporting across trials of proportionate interventions.Trial registrationProspero, CRD42016033781. Registered on 2 February 2016.

Project description:PurposeTo develop a statistical approach that compares patient selection strategies across clinical trials and apply this approach to acute ischemic stroke clinical trials to identify the optimal inclusion criteria.MethodsWe developed a statistical approach that compares the number needed to treat to achieve one success (NNT) along with the number needed to screen to achieve one success (NNS) and assesses if there are significant differences in inclusion criteria, treatment course, and clinical outcome among patients that may have been included/excluded in the trials. We applied this approach to the study population from four recent positive acute stroke clinical trials: MR CLEAN, EXTEND-IA, ESCAPE, and SWIFT PRIME, applying published trial criteria to an independent registry of 612 acute stroke patients, since we did not have access to the complete trial data.ResultsAlthough reported NNT were similar for EXTEND-IA, SWIFT PRIME and ESCAPE, and somewhat higher for MR CLEAN, NNS varied across the trials from 21 for EXTEND-IA, 27 for MR CLEAN, to 46 for ESCAPE and 64 for SWIFT PRIME, reflecting less and more stringent inclusion criteria, respectively. Although there were significant differences in imaging biomarkers and other clinical characteristics among patients that may have been included/excluded in the trials, these differences did not translate to significant differences in treatment course or clinical outcomes.ConclusionsOur study proposes a robust statistical approach that can be applied to a larger pooled trial dataset, if made available, to objectively compare across clinical trials and inform inclusion criteria of future trials. Pooled analysis of the acute stroke trial data is needed to determine which imaging biomarker inclusion criteria are critical and which may be relaxed. If this procedure were applied across the pooled trial data, it could decrease costs and refine the design of future trials to be the most efficacious for the greatest number of patients.

Dataset Information

How to design a pre-specified statistical analysis approach to limit p-hacking in clinical trials: the Pre-SPEC framework.

Publications

How to design a pre-specified statistical analysis approach to limit p-hacking in clinical trials: the Pre-SPEC framework.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets