Limitations of empirical calibration of p-values using observational data.

ABSTRACT: Controversy over non-reproducible published research reporting a statistically significant result has produced substantial discussion in the literature. p-value calibration is a recently proposed procedure for adjusting p-values to account for both random and systematic errors that address one aspect of this problem. The method's validity rests on the key assumption that bias in an effect estimate is drawn from a normal distribution whose mean and variance can be correctly estimated. We investigated the method's control of type I and type II error rates using simulated and real-world data. Under mild violations of underlying assumptions, control of the type I error rate can be conservative, while under more extreme departures, it can be anti-conservative. The extent to which the assumption

SUBMITTER: Gruber S 

PROVIDER: S-EPMC5012943 | biostudies-literature | 2016 Sep

REPOSITORIES: biostudies-literature