Browse
Submit Data
Databases
API
Help

Dataset Information

14 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Developing clinical prediction models when adhering to minimum sample size recommendations: The importance of quantifying bootstrap variability in tuning parameters and predictive performance.

ABSTRACT: Recent minimum sample size formula (Riley et al.) for developing clinical prediction models help ensure that development datasets are of sufficient size to minimise overfitting. While these criteria are known to avoid excessive overfitting on average, the extent of variability in overfitting at recommended sample sizes is unknown. We investigated this through a simulation study and empirical example to develop logistic regression clinical prediction models using unpenalised maximum likelihood estimation, and various post-estimation shrinkage or penalisation methods. While the mean calibration slope was close to the ideal value of one for all methods, penalisation further reduced the level of overfitting, on average, compared to unpenalised methods. This came at the cost of higher variability in predictive performance for penalisation methods in external data. We recommend that penalisation methods are used in data that meet, or surpass, minimum sample size requirements to further mitigate overfitting, and that the variability in predictive performance and any tuning parameters should always be examined as part of the model development process, since this provides additional information over average (optimism-adjusted) performance alone. Lower variability would give reassurance that the developed clinical prediction model will perform well in new individuals from the same population as was used for model development.

SUBMITTER: Martin GP

PROVIDER: S-EPMC8649413 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Similar Datasets

A solution to minimum sample size for regressions.

Project description:Regressions and meta-regressions are widely used to estimate patterns and effect sizes in various disciplines. However, many biological and medical analyses use relatively low sample size (N), contributing to concerns on reproducibility. What is the minimum N to identify the most plausible data pattern using regressions? Statistical power analysis is often used to answer that question, but it has its own problems and logically should follow model selection to first identify the most plausible model. Here we make null, simple linear and quadratic data with different variances and effect sizes. We then sample and use information theoretic model selection to evaluate minimum N for regression models. We also evaluate the use of coefficient of determination (R2) for this purpose; it is widely used but not recommended. With very low variance, both false positives and false negatives occurred at N < 8, but data shape was always clearly identified at N ? 8. With high variance, accurate inference was stable at N ? 25. Those outcomes were consistent at different effect sizes. Akaike Information Criterion weights (AICc wi) were essential to clearly identify patterns (e.g., simple linear vs. null); R2 or adjusted R2 values were not useful. We conclude that a minimum N = 8 is informative given very little variance, but minimum N ? 25 is required for more variance. Alternative models are better compared using information theory indices such as AIC but not R2 or adjusted R2. Insufficient N and R2-based model selection apparently contribute to confusion and low reproducibility in various disciplines. To avoid those problems, we recommend that research based on regressions or meta-regressions use N ? 25.

| S-EPMC7034864 | biostudies-literature

Speeding Up Non-Parametric Bootstrap Computations for Statistics Based on Sample Moments in Small/Moderate Sample Size Applications.

Project description:In this paper we propose a vectorized implementation of the non-parametric bootstrap for statistics based on sample moments. Basically, we adopt the multinomial sampling formulation of the non-parametric bootstrap, and compute bootstrap replications of sample moment statistics by simply weighting the observed data according to multinomial counts instead of evaluating the statistic on a resampled version of the observed data. Using this formulation we can generate a matrix of bootstrap weights and compute the entire vector of bootstrap replications with a few matrix multiplications. Vectorization is particularly important for matrix-oriented programming languages such as R, where matrix/vector calculations tend to be faster than scalar operations implemented in a loop. We illustrate the application of the vectorized implementation in real and simulated data sets, when bootstrapping Pearson's sample correlation coefficient, and compared its performance against two state-of-the-art R implementations of the non-parametric bootstrap, as well as a straightforward one based on a for loop. Our investigations spanned varying sample sizes and number of bootstrap replications. The vectorized bootstrap compared favorably against the state-of-the-art implementations in all cases tested, and was remarkably/considerably faster for small/moderate sample sizes. The same results were observed in the comparison with the straightforward implementation, except for large sample sizes, where the vectorized bootstrap was slightly slower than the straightforward implementation due to increased time expenditures in the generation of weight matrices via multinomial sampling.

| S-EPMC4488363 | biostudies-literature

Quantifying ChIP-seq data: a spiking method providing an internal reference for sample-to-sample normalization.

2014-04-02 | GSE52049 | GEO

Quantifying ChIP-seq data: a spiking method providing an internal reference for sample-to-sample normalization.

Project description:Purpose: Description of a spike-adjusting-method (SAM) to normalize ChIP-seq data . Methods: We performed ChIP-seq of POLR3D and POLR2B with mouse liver supplemented with 2.5% of human DNA. Human DNA will be used as an internal control for ChIP-seq quantification. Results: We show that using the SAM for ChIP-seq quantification improve similarity of POLR3D and POLR2B ChIP-seq replicates samples and improve difference between samples originate from different conditions. Conclusions: The SAM improves comparison of ChIP-seq samples, either by increasing similarity between replicates or by emphasise differences between conditions. Chromatin Immuno-precipitations were performed with antibodies directed against POLR3D (Pol III) and POLR2B (Pol II) using mouse liver material supplemented with human DNA. Immuno-precipitated DNA was next sequenced using Illumina HiSeq. Three different concentrations of human spiked DNA were tested for the Pol III ChIP (2.5%, 5% and 10%). We also sequenced the corresponding inputs (crosslinked DNA from mouse liver). Two concentrations of human spiked DNA (5% and 10%) were tested for the Pol 2 ChIP. We also sequenced the corresponding inputs (crosslinked DNA from mouse liver).

2014-04-02 | E-GEOD-52049 | biostudies-arrayexpress

Bootstrap analysis of diffusion tensor and mean apparent propagator parameters derived from multiband diffusion MRI.

Project description:PurposeTo directly compare diffusion metrics derived from multiband (MB) imaging sequences to those derived using a single-band acquisition.MethodsIn this work, diffusion metrics from DTI and mean apparent propagator MRI derived from a commercial MB sequence with an acceleration factor of 3 are compared with those derived from a conventional diffusion MRI sequence using a novel bootstrapping analysis scheme on oversampled diffusion MRI data. The average parameter values for fractional anisotropy and mean diffusivity derived from DTI, as well as propagator anisotropy and return to origin probability derived from mean apparent propagator MRI, are compared.ResultsFractional anisotropy and propagator anisotropy are very similar when computed from data collected with and without MB, but show minor differences at low and high values of fractional anisotropy/propagator anisotropy. Mean diffusivity values are generally lower in the MB-derived maps, and return to origin probability is generally higher. The coefficient of variation of each parameter is shown to be slightly higher on average from the maps derived from MB versus single band when the TR is short, and slightly lower when the TR of the MB and single-band experiments is equal.ConclusionThese results demonstrate that the MB sequence tested in this work provides very similar results to a conventional diffusion MRI sequence. The MB sequence is affected minimally by the slight decrease in SNR associated with the parallel reconstruction and reduced TR, and there are relaxation effects associated with the reduced TR.

| S-EPMC7041889 | biostudies-literature

Adaptive Tuning Curve Widths Improve Sample Efficient Learning.

Project description:Natural brains perform miraculously well in learning new tasks from a small number of samples, whereas sample efficient learning is still a major open problem in the field of machine learning. Here, we raise the question, how the neural coding scheme affects sample efficiency, and make first progress on this question by proposing and analyzing a learning algorithm that uses a simple reinforce-type plasticity mechanism and does not require any gradients to learn low dimensional mappings. It harnesses three bio-plausible mechanisms, namely, population codes with bell shaped tuning curves, continous attractor mechanisms and probabilistic synapses, to achieve sample efficient learning. We show both theoretically and by simulations that population codes with broadly tuned neurons lead to high sample efficiency, whereas codes with sharply tuned neurons account for high final precision. Moreover, a dynamic adaptation of the tuning width during learning gives rise to both, high sample efficiency and high final precision. We prove a sample efficiency guarantee for our algorithm that lies within a logarithmic factor from the information theoretical optimum. Our simulations show that for low dimensional mappings, our learning algorithm achieves comparable sample efficiency to multi-layer perceptrons trained by gradient descent, although it does not use any gradients. Furthermore, it achieves competitive sample efficiency in low dimensional reinforcement learning tasks. From a machine learning perspective, these findings may inspire novel approaches to improve sample efficiency. From a neuroscience perspective, these findings suggest sample efficiency as a yet unstudied functional role of adaptive tuning curve width.

| S-EPMC7041413 | biostudies-literature

Ad libitum drinking prevents dehydration during physical work in the heat when adhering to occupational heat stress recommendations.

Project description:Government entities issue recommendations that aim to maintain core temperature below 38.0°C and prevent dehydration [>2% body mass loss] in unacclimated workers exposed to heat. Hydration recommendations suggest drinking 237 mL of a cool sport drink every 15-20 min. This is based on the premise that ad libitum drinking results in dehydration due to inadequate fluid replacement, but this has never been examined in the background of recommendation compliant work in the heat. Therefore, we tested the hypothesis that ad libitum drinking results in >2% body mass loss during heat stress recommendation compliant work. Ten subjects completed four trials consisting of 4 hours of exposure to wet bulb globe temperatures (WBGT) of 24.1 ± 0.3°C (A), 26.6 ± 0.2°C (B), 28.5 ± 0.2°C (C), 29.3 ± 0.6°C (D). Subjects walked on a treadmill and work-rest ratios were prescribed as a function of WBGT [work:rest per hour - A: 60:0, B: 45:15, C: 30:30, D: 15:45] and were provided 237 mL of a cool sport drink every 15 min to drink ad libitum. Mean core temperature was higher in Trial A (37.8 ± 0.4°C; p = 0.03) and Trial B (37.6 ± 0.3°C; p = 0.01) versus Trial D (37.3 ± 0.3°C) but did not differ between the other trials (p ≥ 0.20). Body mass loss (A: -0.9 ± 0.7%, B: -0.7 ± 0.5%, C: -0.3 ± 0.5%, D: -0.4 ± 0.6%) was greater in Trial A compared to Trial D (p = 0.04) and was different from 2% body mass loss in all trials (p ≤ 0.01). Ad libitum drinking during recommendation compliant work in the heat rarely resulted in dehydration. Registered Clinical Trial (NCT04767347).

| S-EPMC9542357 | biostudies-literature

Minimum Reporting Standards for in vivo Magnetic Resonance Spectroscopy (MRSinMRS): Experts' consensus recommendations.

Project description:The translation of MRS to clinical practice has been impeded by the lack of technical standardization. There are multiple methods of acquisition, post-processing, and analysis whose details greatly impact the interpretation of the results. These details are often not fully reported, making it difficult to assess MRS studies on a standardized basis. This hampers the reviewing of manuscripts, limits the reproducibility of study results, and complicates meta-analysis of the literature. In this paper a consensus group of MRS experts provides minimum guidelines for the reporting of MRS methods and results, including the standardized description of MRS hardware, data acquisition, analysis, and quality assessment. This consensus statement describes each of these requirements in detail and includes a checklist to assist authors and journal reviewers and to provide a practical way for journal editors to ensure that MRS studies are reported in full.

| S-EPMC8647919 | biostudies-literature

Quantifying the Importance of Rapid Adjustments for Global Precipitation Changes.

Project description:Different climate drivers influence precipitation in different ways. Here we use radiative kernels to understand the influence of rapid adjustment processes on precipitation in climate models. Rapid adjustments are generally triggered by the initial heating or cooling of the atmosphere from an external climate driver. For precipitation changes, rapid adjustments due to changes in temperature, water vapor, and clouds are most important. In this study we have investigated five climate drivers (CO2, CH4, solar irradiance, black carbon, and sulfate aerosols). The fast precipitation responses to a doubling of CO2 and a 10-fold increase in black carbon are found to be similar, despite very different instantaneous changes in the radiative cooling, individual rapid adjustments, and sensible heating. The model diversity in rapid adjustments is smaller for the experiment involving an increase in the solar irradiance compared to the other climate driver perturbations, and this is also seen in the precipitation changes.

| S-EPMC6360531 | biostudies-literature

Neglected Importance of Anharmonicity in Quantifying the Renner-Teller Effect.

Project description:Linear molecules in degenerate electronic states are influenced by the Renner-Teller (RT) effect. Currently the formula to quantify this vibronic interaction only considers the harmonic term. However, such a harmonic formula cannot well describe molecules that have double-well potential along the bending coordinate. In this work, we propose a new formula to quantify the RT effect by explicitly including the anharmonic term. A sign function and a delta function are additionally included to distinguish different scenarios. It is mathematically proved that the new formula is capable of differentiating different degrees of RT splitting. Representative molecules of different types are calculated. According to the new formula, molecules experiencing a weak RT effect have a positive parameter smaller than 1. For those experiencing a medium RT effect, the parameter will be negative. For others experiencing a large RT effect, the parameter will be larger than 1.

| S-EPMC9730495 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data