Dataset Information

Predicting sample size required for classification performance.

ABSTRACT:

Background

Supervised learning methods need annotated data in order to generate efficient models. Annotated data, however, is a relatively scarce resource and can be expensive to obtain. For both passive and active learning methods, there is a need to estimate the size of the annotated sample required to reach a performance target.

Methods

We designed and implemented a method that fits an inverse power law model to points of a given learning curve created using a small annotated training set. Fitting is carried out using nonlinear weighted least squares optimization. The fitted model is then used to predict the classifier's performance and confidence interval for larger sample sizes. For evaluation, the nonlinear weighted curve fitting method was applied to a set of learning curves generated using clinical text and waveform classification tasks with active and passive sampling methods, and predictions were validated using standard goodness of fit measures. As control we used an un-weighted fitting method.

Results

A total of 568 models were fitted and the model predictions were compared with the observed performances. Depending on the data set and sampling method, it took between 80 to 560 annotated samples to achieve mean average and root mean squared error below 0.01. Results also show that our weighted fitting method outperformed the baseline un-weighted method (p < 0.05).

Conclusions

This paper describes a simple and effective sample size prediction algorithm that conducts weighted fitting of learning curves. The algorithm outperformed an un-weighted algorithm described in previous literature. It can help researchers determine annotation sample size for supervised machine learning.

SUBMITTER: Figueroa RL

PROVIDER: S-EPMC3307431 | biostudies-literature | 2012 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Predicting sample size required for classification performance.

Figueroa Rosa L RL Zeng-Treitler Qing Q Kandula Sasikiran S Ngo Long H LH

BMC medical informatics and decision making 20120215

<h4>Background</h4>Supervised learning methods need annotated data in order to generate efficient models. Annotated data, however, is a relatively scarce resource and can be expensive to obtain. For both passive and active learning methods, there is a need to estimate the size of the annotated sample required to reach a performance target.<h4>Methods</h4>We designed and implemented a method that fits an inverse power law model to points of a given learning curve created using a small annotated t ...[more]

PMID: 22336388

Similar Datasets

Project description:BACKGROUND: Supervised learning for classification of cancer employs a set of design examples to learn how to discriminate between tumors. In practice it is crucial to confirm that the classifier is robust with good generalization performance to new examples, or at least that it performs better than random guessing. A suggested alternative is to obtain a confidence interval of the error rate using repeated design and test sets selected from available examples. However, it is known that even in the ideal situation of repeated designs and tests with completely novel samples in each cycle, a small test set size leads to a large bias in the estimate of the true variance between design sets. Therefore different methods for small sample performance estimation such as a recently proposed procedure called Repeated Random Sampling (RSS) is also expected to result in heavily biased estimates, which in turn translates into biased confidence intervals. Here we explore such biases and develop a refined algorithm called Repeated Independent Design and Test (RIDT). RESULTS: Our simulations reveal that repeated designs and tests based on resampling in a fixed bag of samples yield a biased variance estimate. We also demonstrate that it is possible to obtain an improved variance estimate by means of a procedure that explicitly models how this bias depends on the number of samples used for testing. For the special case of repeated designs and tests using new samples for each design and test, we present an exact analytical expression for how the expected value of the bias decreases with the size of the test set. CONCLUSION: We show that via modeling and subsequent reduction of the small sample bias, it is possible to obtain an improved estimate of the variance of classifier performance between design sets. However, the uncertainty of the variance estimate is large in the simulations performed indicating that the method in its present form cannot be directly applied to small data sets.

Project description:Risk-prediction models for health outcomes are used in practice as part of clinical decision-making, and it is essential that their performance be externally validated. An important aspect in the design of a validation study is choosing an adequate sample size. In this paper, we investigate the sample size requirements for validation studies with binary outcomes to estimate measures of predictive performance (C-statistic for discrimination and calibration slope and calibration in the large). We aim for sufficient precision in the estimated measures. In addition, we investigate the sample size to achieve sufficient power to detect a difference from a target value. Under normality assumptions on the distribution of the linear predictor, we obtain simple estimators for sample size calculations based on the measures above. Simulation studies show that the estimators perform well for common values of the C-statistic and outcome prevalence when the linear predictor is marginally Normal. Their performance deteriorates only slightly when the normality assumptions are violated. We also propose estimators which do not require normality assumptions but require specification of the marginal distribution of the linear predictor and require the use of numerical integration. These estimators were also seen to perform very well under marginal normality. Our sample size equations require a specified standard error (SE) and the anticipated C-statistic and outcome prevalence. The sample size requirement varies according to the prognostic strength of the model, outcome prevalence, choice of the performance measure and study objective. For example, to achieve an SE < 0.025 for the C-statistic, 60-170 events are required if the true C-statistic and outcome prevalence are between 0.64-0.85 and 0.05-0.3, respectively. For the calibration slope and calibration in the large, achieving SE < 0.15 would require 40-280 and 50-100 events, respectively. Our estimators may also be used for survival outcomes when the proportion of censored observations is high.

Project description:BackgroundAs an immunogenic cancer, crosstalk between cancer cells and immune cells has been gradually recognized in bladder cancer (BC). Several studies have emphasized the clinical significance of the molecular stratification of BC without highlighting the role of the immune microenvironment. Although immunotherapy acted as a prospective treatment, more precise molecular stratification should be established to select those sensitive to immunotherapy.MethodsTo select specific immune genes forming subtypes indicating disparate prognoses, we performed bioinformatic analysis using BC transcriptomic profiles from six published datasets, with 408 BC samples in The Cancer Genome Atlas (TCGA) database and 295 individuals in International Cancer Genome Consortium (ICGC) database. Survival analyses were conducted using Kaplan-Meier curves, while Kruskal-Wallis tests were applied to test the differences among groups. Except for unsupervised clustering based on the differential expression of genes, we additionally performed binomial logistic regression, focusing on the mRNA level of a single sample.ResultsUnsupervised clustering showed that 4 clusters captured the best segmentation. After validation with survival data and simplification using binomial logistic regression, we found that cluster B and cluster D showed worse survival outcomes (P=0.012). Considering the similar survival outcomes of these two clusters, we recombined and performed another survival analysis, which also showed significant survival differences (P=0.0041). Bonding with clinical data, a greater proportion of risk factors were assigned to the worse prognosis subtype, especially showing higher grades in the subtype (P<0.001). In addition, immune cell infiltration, single nucleotide polymorphism (SNP) and copy number variation (CNV) all showed differences between clusters, indicating changes in the immune microenvironment and mutation burden. Through phenotypical analysis, we found metabolism and proliferation phenotypes associated with the immune clusters and mutually exclusive in BC, of which proliferation contributed to worse outcomes. Using the tumor immune dysfunction and exclusion (TIDE) score, a worse immunotherapy benefit was predicted in clusters B&D, defined as the worse prognosis subtype.ConclusionsWith this novel clustering criterion based on immune-related genes, we provide a better understanding of the immune microenvironment, further guiding the use of immunotherapy.

Dataset Information

Predicting sample size required for classification performance.

Background

Methods

Results

Conclusions

Publications

Predicting sample size required for classification performance.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets