The calibrated model-based concordance improved assessment of discriminative ability in patient clusters of limited sample size.
Ontology highlight
ABSTRACT: Background
Discriminative ability is an important aspect of prediction model performance, but challenging to assess in clustered (e.g., multicenter) data. Concordance (c)-indexes may be too extreme within small clusters. We aimed to define a new approach for the assessment of discriminative ability in clustered data.Methods
We assessed discriminative ability of a prediction model for the binary outcome mortality after traumatic brain injury within centers of the CRASH trial. With multilevel logistic regression analysis, we estimated cluster-specific calibration slopes which we used to obtain the recently proposed calibrated model-based concordance (c-mbc) within each cluster. We compared the c-mbc with the naïve c-index in centers of the CRASH trial and in simulations of clusters with varying calibration slopes.Results
The c-mbc was less extreme in distribution than the c-index in 19 European centers (internal validation; n = 1716) and 36 non-European centers (external validation; n = 3135) of the CRASH trial. In simulations, the c-mbc was biased but less variable than the naïve c-index, resulting in lower root mean squared errors.Conclusions
The c-mbc, based on multilevel regression analysis of the calibration slope, is an attractive alternative to the c-index as a measure of discriminative ability in multicenter studies with patient clusters of limited sample size.
SUBMITTER: van Klaveren D
PROVIDER: S-EPMC6551913 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Advances in neuroimaging, genomic, motion tracking, eye-tracking and many other technology-based data collection methods have led to a torrent of high dimensional datasets, which commonly have a small number of samples because of the intrinsic high cost of data collection involving human participants. High dimensional data with a small number of samples is of critical importance for identifying biomarkers and conducting feasibility and pilot work, however it can lead to biased machine learning (ML) performance estimates. Our review of studies which have applied ML to predict autistic from non-autistic individuals showed that small sample size is associated with higher reported classification accuracy. Thus, we have investigated whether this bias could be caused by the use of validation methods which do not sufficiently control overfitting. Our simulations show that K-fold Cross-Validation (CV) produces strongly biased performance estimates with small sample sizes, and the bias is still evident with sample size of 1000. Nested CV and train/test split approaches produce robust and unbiased performance estimates regardless of sample size. We also show that feature selection if performed on pooled training and testing data is contributing to bias considerably more than parameter tuning. In addition, the contribution to bias by data dimensionality, hyper-parameter space and number of CV folds was explored, and validation methods were compared with discriminable data. The results suggest how to design robust testing methodologies when working with small datasets and how to interpret the results of other studies based on what validation method was used.
| S-EPMC6837442 | biostudies-literature
Project description:BackgroundWe describe a novel strategy for power and sample size determination developed for studies utilizing investigational technologies with limited available preliminary data, specifically of imaging biomarkers. We evaluated diffuse optical spectroscopic imaging (DOSI), an experimental noninvasive imaging technique that may be capable of assessing changes in mammographic density. Because there is significant evidence that tamoxifen treatment is more effective at reducing breast cancer risk when accompanied by a reduction of breast density, we designed a study to assess the changes from baseline in DOSI imaging biomarkers that may reflect fluctuations in breast density in premenopausal women receiving tamoxifen.MethodWhile preliminary data demonstrate that DOSI is sensitive to mammographic density in women about to receive neoadjuvant chemotherapy for breast cancer, there is no information on DOSI in tamoxifen treatment. Since the relationship between magnetic resonance imaging (MRI) and DOSI has been established in previous studies, we developed a statistical simulation approach utilizing information from an investigation of MRI assessment of breast density in 16 women before and after treatment with tamoxifen to estimate the changes in DOSI biomarkers due to tamoxifen.ResultsThree sets of 10,000 pairs of MRI breast density data with correlation coefficients of 0.5, 0.8 and 0.9 were simulated and generated and were used to simulate and generate a corresponding 5,000,000 pairs of DOSI values representing water, ctHHB, and lipid. Minimum sample sizes needed per group for specified clinically-relevant effect sizes were obtained.ConclusionThe simulation techniques we describe can be applied in studies of other experimental technologies to obtain the important preliminary data to inform the power and sample size calculations.
| S-EPMC5406943 | biostudies-literature
Project description:In bird species where males incubate but are smaller than females, egg size may be constrained by male body size, and hence ability to incubate the eggs. Using data from 71 such shorebird species, we show that egg size decreases as the degree of female-biased sexual size dimorphism increases, after controlling for female body mass. Relative egg size was not related to mean clutch size. However, when controlling for mating system, the relationship between female-biased sexual size dimorphism and relative egg size was only significant in polyandrous species. The relatively small eggs of socially polyandrous shorebirds have previously been explained as an energy-saving strategy associated with the production of multiple clutches. Our findings suggest that egg size evolution is better explained by male incubation limitation in these birds.
| S-EPMC1618901 | biostudies-literature
Project description:BACKGROUND: Microarray technology is often used to identify the genes that are differentially expressed between two biological conditions. On the other hand, since microarray datasets contain a small number of samples and a large number of genes, it is usually desirable to identify small gene subsets with distinct pattern between sample classes. Such gene subsets are highly discriminative in phenotype classification because of their tightly coupling features. Unfortunately, such identified classifiers usually tend to have poor generalization properties on the test samples due to overfitting problem. RESULTS: We propose a novel approach combining both supervised learning with unsupervised learning techniques to generate increasingly discriminative gene clusters in an iterative manner. Our experiments on both simulated and real datasets show that our method can produce a series of robust gene clusters with good classification performance compared with existing approaches. CONCLUSION: This backward approach for refining a series of highly discriminative gene clusters for classification purpose proves to be very consistent and stable when applied to various types of training samples.
| S-EPMC2559882 | biostudies-other
Project description:PurposeThe objective of this study was to expand the international psychometric validation of the Cancer Communication Assessment Tool for Patients and Families (CCAT-PF) within a sample of Australian cancer patients.MethodsSurvey data from 181 cancer patient-caregiver dyads ≥ 18 years of age with solid or haematological cancers were analysed (85.4% response rate). Spearman's rho was used to examine the correlation between CCAT-P and CCAT-F scores and weighted kappa the agreement between them. Exploratory factor analysis using scree plot and Kaiser-Guttman criteria was conducted to evaluate the scale structure. Cronbach's α and Pearson correlation coefficients were used to measure internal consistency and concurrent validity respectively.ResultsMean scores were the following: CCAT-P 46.2 (9.8), CCAT-F 45.7 (9.4), and CCAT-PF 24.1 (8.0). We confirmed the poor concordance between patient and caregiver reporting of items in the CCAT-PF, with all but two items having weighted kappa values < 0.20 and Spearman's rho < 0.19. We derived a three-factor solution, disclosure, limitation of treatment, and treatment decision making, with reliability ranging from Cronbach's α = 0.43-0.53. The CCAT-P and CCAT-F showed strong correlations with preparation for decision-making (CCAT-P: r = 0.0.92; CCATF: r = 0.0.93) but were weakly associated with patient/caregiver distress related with having difficult conversations on future care planning.ConclusionPreliminary validation of the CCAT-PF in the Australian setting has shown some similar psychometric properties to previously published studies, further supporting its potential utility as a tool to assess patient-caregiver dyadic communication.Trial registrationACTRN12620001035910 12/10/2020 retrospectively registered.
| S-EPMC9385757 | biostudies-literature
Project description:Co-expression has been widely used to identify novel regulatory relationships using high throughput measurements, such as microarray and RNA-seq data. Evaluation studies on co-expression network analysis methods mostly focus on networks of small or medium size of up to a few hundred nodes. For large networks, simulated expression data usually consist of hundreds or thousands of profiles with different perturbations or knock-outs, which is uncommon in real experiments due to their cost and the amount of work required. Thus, the performances of co-expression network analysis methods on large co-expression networks consisting of a few thousand nodes, with only a small number of profiles with a single perturbation, which more accurately reflect normal experimental conditions, are generally uncharacterized and unknown.We proposed a novel network inference methods based on Relevance Low order Partial Correlation (RLowPC). RLowPC method uses a two-step approach to select on the high-confidence edges first by reducing the search space by only picking the top ranked genes from an intial partial correlation analysis and, then computes the partial correlations in the confined search space by only removing the linear dependencies from the shared neighbours, largely ignoring the genes showing lower association.We selected six co-expression-based methods with good performance in evaluation studies from the literature: Partial correlation, PCIT, ARACNE, MRNET, MRNETB and CLR. The evaluation of these methods was carried out on simulated time-series data with various network sizes ranging from 100 to 3000 nodes. Simulation results show low precision and recall for all of the above methods for large networks with a small number of expression profiles. We improved the inference significantly by refinement of the top weighted edges in the pre-inferred partial correlation networks using RLowPC. We found improved performance by partitioning large networks into smaller co-expressed modules when assessing the method performance within these modules.The evaluation results show that current methods suffer from low precision and recall for large co-expression networks where only a small number of profiles are available. The proposed RLowPC method effectively reduces the indirect edges predicted as regulatory relationships and increases the precision of top ranked predictions. Partitioning large networks into smaller highly co-expressed modules also helps to improve the performance of network inference methods. The RLowPC R package for network construction, refinement and evaluation is available at GitHub: https://github.com/wyguo/RLowPC .
| S-EPMC5477119 | biostudies-literature
Project description:Although acquisition of leukemia-associated somatic mutations by 1 or more hematopoietic stem cells is inevitable with advancing age, its consequences are highly variable, ranging from clinically silent clonal hematopoiesis (CH) to leukemic progression. To investigate the influence of heritable factors on CH, we performed deep targeted sequencing of blood DNA from 52 monozygotic (MZ) and 27 dizygotic (DZ) twin pairs (aged 70-99 years). Using this highly sensitive approach, we identified CH (variant allele frequency ≥0.5%) in 62% of individuals. We did not observe higher concordance for CH within MZ twin pairs as compared with that within DZ twin pairs, or to that expected by chance. However, we did identify 2 MZ pairs in which both twins harbored identical rare somatic mutations, suggesting a shared cell of origin. Finally, in 3 MZ twin pairs harboring mutations in the same driver genes, serial blood samples taken 4 to 5 years apart showed substantial twin-to-twin variability in clonal trajectories. Our findings propose that the inherited genome does not exert a dominant influence on the behavior of adult CH and provide evidence that CH mutations may be acquired in utero.
| S-EPMC6978156 | biostudies-literature
Project description:Though many people with multiple sclerosis (MS) have comorbidities, the use of generic and disease-specific health related quality of life (HRQOL) scales to discriminate the effects of comorbidity has not been established. The utility of these scales to discriminate differences between persons with varying levels of disability is also unknown.Using online questionnaires, a convenience sample of Albertans with MS was recruited between July 2011 and March 2013. Participants completed demographic questions, a validated comorbidity questionnaire, a self-reported disability scale, and the following HRQOL scales: the Short Form (SF)-36, SF-6D, Health Utilities Index-Mark III (HUI-III), and Multiple Sclerosis Quality of Life-54 (MSQOL-54). The ability of each HRQOL scale to distinguish between comorbidity groups was assessed using a one-way analysis of covariance, adjusting for age, sex, disease course, and disability level.Five hundred sixty three participants completed all relevant questionnaires. All HRQOL measures distinguished between persons with or without depression, while none were able to distinguish between participants with or without hypertension, thyroid disease, irritable bowel syndrome, or osteoporosis. The SF-36 physical scale, SF-6D, HUI-III, and MSQOL-54 physical scales were able to distinguish between all disability groups, though the HUI-III was better able to distinguish between individuals with moderate versus severe disability.Disease-specific measures would discriminate better between those with and without comorbidities than generic-specific measures and the HUI-III would discriminate best between persons with differing severities of disability. Generic or disease-specific measures may be useful in future studies examining the effects of comorbidity in MS and the effects of treatment of comorbidities in MS.
| S-EPMC5740906 | biostudies-other
Project description:BACKGROUND: Supervised learning for classification of cancer employs a set of design examples to learn how to discriminate between tumors. In practice it is crucial to confirm that the classifier is robust with good generalization performance to new examples, or at least that it performs better than random guessing. A suggested alternative is to obtain a confidence interval of the error rate using repeated design and test sets selected from available examples. However, it is known that even in the ideal situation of repeated designs and tests with completely novel samples in each cycle, a small test set size leads to a large bias in the estimate of the true variance between design sets. Therefore different methods for small sample performance estimation such as a recently proposed procedure called Repeated Random Sampling (RSS) is also expected to result in heavily biased estimates, which in turn translates into biased confidence intervals. Here we explore such biases and develop a refined algorithm called Repeated Independent Design and Test (RIDT). RESULTS: Our simulations reveal that repeated designs and tests based on resampling in a fixed bag of samples yield a biased variance estimate. We also demonstrate that it is possible to obtain an improved variance estimate by means of a procedure that explicitly models how this bias depends on the number of samples used for testing. For the special case of repeated designs and tests using new samples for each design and test, we present an exact analytical expression for how the expected value of the bias decreases with the size of the test set. CONCLUSION: We show that via modeling and subsequent reduction of the small sample bias, it is possible to obtain an improved estimate of the variance of classifier performance between design sets. However, the uncertainty of the variance estimate is large in the simulations performed indicating that the method in its present form cannot be directly applied to small data sets.
| S-EPMC1435937 | biostudies-literature
Project description:Various neuroimaging measures are being evaluated for tracking Alzheimer's disease (AD) progression in therapeutic trials, including measures of structural brain change based on repeated scanning of patients with magnetic resonance imaging (MRI). Methods to compute brain change must be robust to scan quality. Biases may arise if any scans are thrown out, as this can lead to the true changes being overestimated or underestimated. Here we analyzed the full MRI dataset from the first phase of Alzheimer's Disease Neuroimaging Initiative (ADNI-1) from the first phase of Alzheimer's Disease Neuroimaging Initiative (ADNI-1) and assessed several sources of bias that can arise when tracking brain changes with structural brain imaging methods, as part of a pipeline for tensor-based morphometry (TBM). In all healthy subjects who completed MRI scanning at screening, 6, 12, and 24months, brain atrophy was essentially linear with no detectable bias in longitudinal measures. In power analyses for clinical trials based on these change measures, only 39AD patients and 95 mild cognitive impairment (MCI) subjects were needed for a 24-month trial to detect a 25% reduction in the average rate of change using a two-sided test (?=0.05, power=80%). Further sample size reductions were achieved by stratifying the data into Apolipoprotein E (ApoE) ?4 carriers versus non-carriers. We show how selective data exclusion affects sample size estimates, motivating an objective comparison of different analysis techniques based on statistical power and robustness. TBM is an unbiased, robust, high-throughput imaging surrogate marker for large, multi-site neuroimaging studies and clinical trials of AD and MCI.
| S-EPMC3785376 | biostudies-literature