Dataset Information

Bias and variance reduction in estimating the proportion of true-null hypotheses.

ABSTRACT: When testing a large number of hypotheses, estimating the proportion of true nulls, denoted by ?(0), becomes increasingly important. This quantity has many applications in practice. For instance, a reliable estimate of ?(0) can eliminate the conservative bias of the Benjamini-Hochberg procedure on controlling the false discovery rate. It is known that most methods in the literature for estimating ?(0) are conservative. Recently, some attempts have been paid to reduce such estimation bias. Nevertheless, they are either over bias corrected or suffering from an unacceptably large estimation variance. In this paper, we propose a new method for estimating ?(0) that aims to reduce the bias and variance of the estimation simultaneously. To achieve this, we first utilize the probability density functions of false-null p-values and then propose a novel algorithm to estimate the quantity of ?(0). The statistical behavior of the proposed estimator is also investigated. Finally, we carry out extensive simulation studies and several real data analysis to evaluate the performance of the proposed estimator. Both simulated and real data demonstrate that the proposed method may improve the existing literature significantly.

SUBMITTER: Cheng Y

PROVIDER: S-EPMC4263223 | biostudies-literature | 2015 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Bias and variance reduction in estimating the proportion of true-null hypotheses.

Cheng Yebin Y Gao Dexiang D Tong Tiejun T

Biostatistics (Oxford, England) 20140623 1

When testing a large number of hypotheses, estimating the proportion of true nulls, denoted by π(0), becomes increasingly important. This quantity has many applications in practice. For instance, a reliable estimate of π(0) can eliminate the conservative bias of the Benjamini-Hochberg procedure on controlling the false discovery rate. It is known that most methods in the literature for estimating π(0) are conservative. Recently, some attempts have been paid to reduce such estimation bias. Nevert ...[more]

PMID: 24963010

Similar Datasets

Project description:BackgroundCholera surveillance relies on clinical diagnosis of acute watery diarrhea. Suspected cholera case definitions have high sensitivity but low specificity, challenging our ability to characterize cholera burden and epidemiology. Our objective was to estimate the proportion of clinically suspected cholera that are true Vibrio cholerae infections and identify factors that explain variation in positivity.Methods and findingsWe conducted a systematic review of studies that tested ≥10 suspected cholera cases for V. cholerae O1/O139 using culture, PCR, and/or a rapid diagnostic test. We searched PubMed, Embase, Scopus, and Google Scholar for studies that sampled at least one suspected case between January 1, 2000 and April 19, 2023, to reflect contemporary patterns in V. cholerae positivity. We estimated diagnostic test sensitivity and specificity using a latent class meta-analysis. We estimated V. cholerae positivity using a random-effects meta-analysis, adjusting for test performance. We included 119 studies from 30 countries. V. cholerae positivity was lower in studies with representative sampling and in studies that set minimum ages in suspected case definitions. After adjusting for test performance, on average, 52% (95% credible interval (CrI): 24%, 80%) of suspected cases represented true V. cholerae infections. After adjusting for test performance and study methodology, the odds of a suspected case having a true infection were 5.71 (odds ratio 95% CrI: 1.53, 15.43) times higher when surveillance was initiated in response to an outbreak than in non-outbreak settings. Variation across studies was high, and a limitation of our approach was that we were unable to explain all the heterogeneity with study-level attributes, including diagnostic test used, setting, and case definitions.ConclusionsIn this study, we found that burden estimates based on suspected cases alone may overestimate the incidence of medically attended cholera by 2-fold. However, accounting for cases missed by traditional clinical surveillance is key to unbiased cholera burden estimates. Given the substantial variability in positivity between settings, extrapolations from suspected to confirmed cases, which is necessary to estimate cholera incidence rates without exhaustive testing, should be based on local data.

Project description:BACKGROUND: Supervised learning for classification of cancer employs a set of design examples to learn how to discriminate between tumors. In practice it is crucial to confirm that the classifier is robust with good generalization performance to new examples, or at least that it performs better than random guessing. A suggested alternative is to obtain a confidence interval of the error rate using repeated design and test sets selected from available examples. However, it is known that even in the ideal situation of repeated designs and tests with completely novel samples in each cycle, a small test set size leads to a large bias in the estimate of the true variance between design sets. Therefore different methods for small sample performance estimation such as a recently proposed procedure called Repeated Random Sampling (RSS) is also expected to result in heavily biased estimates, which in turn translates into biased confidence intervals. Here we explore such biases and develop a refined algorithm called Repeated Independent Design and Test (RIDT). RESULTS: Our simulations reveal that repeated designs and tests based on resampling in a fixed bag of samples yield a biased variance estimate. We also demonstrate that it is possible to obtain an improved variance estimate by means of a procedure that explicitly models how this bias depends on the number of samples used for testing. For the special case of repeated designs and tests using new samples for each design and test, we present an exact analytical expression for how the expected value of the bias decreases with the size of the test set. CONCLUSION: We show that via modeling and subsequent reduction of the small sample bias, it is possible to obtain an improved estimate of the variance of classifier performance between design sets. However, the uncertainty of the variance estimate is large in the simulations performed indicating that the method in its present form cannot be directly applied to small data sets.

Dataset Information

Bias and variance reduction in estimating the proportion of true-null hypotheses.

Publications

Bias and variance reduction in estimating the proportion of true-null hypotheses.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets