Dataset Information

Bias and variance reduction in estimating the proportion of true-null hypotheses.

ABSTRACT: When testing a large number of hypotheses, estimating the proportion of true nulls, denoted by ?(0), becomes increasingly important. This quantity has many applications in practice. For instance, a reliable estimate of ?(0) can eliminate the conservative bias of the Benjamini-Hochberg procedure on controlling the false discovery rate. It is known that most methods in the literature for estimating ?(0) are conservative. Recently, some attempts have been paid to reduce such estimation bias. Nevertheless, they are either over bias corrected or suffering from an unacceptably large estimation variance. In this paper, we propose a new method for estimating ?(0) that aims to reduce the bias and variance of the estimation simultaneously. To achieve this, we first utilize the probability density functions of false-null p-values and then propose a novel algorithm to estimate the quantity of ?(0). The statistical behavior of the proposed estimator is also investigated. Finally, we carry out extensive simulation studies and several real data analysis to evaluate the performance of the proposed estimator. Both simulated and real data demonstrate that the proposed method may improve the existing literature significantly.

SUBMITTER: Cheng Y

PROVIDER: S-EPMC4263223 | biostudies-literature | 2015 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Bias and variance reduction in estimating the proportion of true-null hypotheses.

Cheng Yebin Y Gao Dexiang D Tong Tiejun T

Biostatistics (Oxford, England) 20140623 1

When testing a large number of hypotheses, estimating the proportion of true nulls, denoted by π(0), becomes increasingly important. This quantity has many applications in practice. For instance, a reliable estimate of π(0) can eliminate the conservative bias of the Benjamini-Hochberg procedure on controlling the false discovery rate. It is known that most methods in the literature for estimating π(0) are conservative. Recently, some attempts have been paid to reduce such estimation bias. Nevert ...[more]

PMID: 24963010

Similar Datasets

Project description:BACKGROUND: Supervised learning for classification of cancer employs a set of design examples to learn how to discriminate between tumors. In practice it is crucial to confirm that the classifier is robust with good generalization performance to new examples, or at least that it performs better than random guessing. A suggested alternative is to obtain a confidence interval of the error rate using repeated design and test sets selected from available examples. However, it is known that even in the ideal situation of repeated designs and tests with completely novel samples in each cycle, a small test set size leads to a large bias in the estimate of the true variance between design sets. Therefore different methods for small sample performance estimation such as a recently proposed procedure called Repeated Random Sampling (RSS) is also expected to result in heavily biased estimates, which in turn translates into biased confidence intervals. Here we explore such biases and develop a refined algorithm called Repeated Independent Design and Test (RIDT). RESULTS: Our simulations reveal that repeated designs and tests based on resampling in a fixed bag of samples yield a biased variance estimate. We also demonstrate that it is possible to obtain an improved variance estimate by means of a procedure that explicitly models how this bias depends on the number of samples used for testing. For the special case of repeated designs and tests using new samples for each design and test, we present an exact analytical expression for how the expected value of the bias decreases with the size of the test set. CONCLUSION: We show that via modeling and subsequent reduction of the small sample bias, it is possible to obtain an improved estimate of the variance of classifier performance between design sets. However, the uncertainty of the variance estimate is large in the simulations performed indicating that the method in its present form cannot be directly applied to small data sets.

Dataset Information

Bias and variance reduction in estimating the proportion of true-null hypotheses.

Publications

Bias and variance reduction in estimating the proportion of true-null hypotheses.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure