Dataset Information

Don't lose samples to estimation.

ABSTRACT: In a typical predictive modeling task, we are asked to produce a final predictive model to employ operationally for predictions, as well as an estimate of its out-of-sample predictive performance. Typically, analysts hold out a portion of the available data, called a Test set, to estimate the model predictive performance on unseen (out-of-sample) records, thus "losing these samples to estimation." However, this practice is unacceptable when the total sample size is low. To avoid losing data to estimation, we need a shift in our perspective: we do not estimate the performance of a specific model instance; we estimate the performance of the pipeline that produces the model. This pipeline is applied on all available samples to produce the final model; no samples are lost to estimation. An estimate of its performance is provided by training the same pipeline on subsets of the samples. When multiple pipelines are tried, additional considerations that correct for the "winner's curse" need to be in place.

SUBMITTER: Tsamardinos I

PROVIDER: S-EPMC9782254 | biostudies-literature | 2022 Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Don't lose samples to estimation.

Tsamardinos Ioannis I

Patterns (New York, N.Y.) 20221209 12

In a typical predictive modeling task, we are asked to produce a final predictive model to employ operationally for predictions, as well as an estimate of its out-of-sample predictive performance. Typically, analysts hold out a portion of the available data, called a Test set, to estimate the model predictive performance on unseen (out-of-sample) records, thus "losing these samples to estimation." However, this practice is unacceptable when the total sample size is low. To avoid losing data to e ...[more]

PMID: 36569551

Similar Datasets

Project description:BackgroundCommunity-driven invasion, also known as community coalescence, occurs widely in natural ecosystems. Despite that, our knowledge about the process and mechanisms controlling community-driven invasion in soil ecosystems is lacking. Here, we performed a set of coalescence experiments in soil microcosms and assessed impacts up to 60 days after coalescence by quantifying multiple traits (compositional, functional, and metabolic) of the invasive and coalescent communities.ResultsOur results showed that coalescences significantly triggered changes in the resident community's succession trajectory and functionality (carbohydrate metabolism), even when the size of the invasive community is small (~ 5% of the resident density) and 99% of the invaders failed to survive. The invasion impact was mainly due to the high suppression of constant residents (65% on average), leading to a lose-lose situation where both invaders and residents suffered with coalescence. Our results showed that surviving residents could benefit from the coalescence, which supports the theory of "competition-driven niche segregation" at the microbial community level. Furthermore, the result showed that both short- and long-term coalescence effects were predicted by similarity and unevenness indexes of compositional, functional, and metabolic traits of invasive communities. This indicates the power of multi-level traits in monitoring microbial community succession. In contrast, the varied importance of different levels of traits suggests that competitive processes depend on the composition of the invasive community.ConclusionsOur results shed light on the process and consequence of community coalescences and highlight that resource competition between invaders and residents plays a critical role in soil microbial community coalescences. These findings provide valuable insights for understanding and predicting soil microbial community succession in frequently disturbed natural and agroecosystems. Video Abstract.

Project description:BACKGROUND:Diffuse axonal injury (DAI) on magnetic resonance imaging has been associated with poor functional outcome after moderate-severe traumatic brain injury (msTBI). Yet, DAI assessment with highly sensitive magnetic resonance imaging techniques is unfeasible in the acute trauma setting, and computed tomography (CT) remains the key diagnostic modality despite its lower sensitivity. We sought to determine whether CT-defined hemorrhagic DAI (hDAI) is associated with discharge and favorable 3- and 12-month functional outcome (Glasgow Coma Scale score ≥4) after msTBI. METHODS:We analyzed 361 msTBI patients from the single-center longitudinal Outcome Prognostication in Traumatic Brain Injury study collected over 6 years (November 2009 to November 2015) with prospective outcome assessments at 3 months and 12 months. Patients with microhemorrhages on CT were designated "CT-hDAI-positive" and those without as "CT-hDAI-negative." For secondary analyses "CT-hDAI-positive" was stratified into two phenotypes according to presence ("associated") versus absence ("predominant") of concomitant large acute traumatic lesions to determine whether presence versus absence of additional focal mass lesions portends a different prognosis. RESULTS:Seventy (19%) patients were CT-hDAI-positive (n = 36 predominant; n = 34 associated hDAI). In univariate analyses, CT-hDAI-positive status was associated with discharge survival (p = 0.004) and favorable outcome at 3 months (p = 0.003) and 12 months (p = 0.005). After multivariable adjustment, CT-hDAI positivity was no longer associated with discharge survival and functional outcome (all ps > 0.05). Stratified by hDAI phenotype, predominant hDAI patients had worse trauma severity, longer intensive care unit stays, and more systemic medical complications. Predominant hDAI, but not associated hDAI, was an independent predictor of discharge survival (adjusted odds ratio, 24.7; 95% confidence interval [CI], 3.2-192.6; p = 0.002) and favorable 12-month outcome (adjusted odds ratio, 4.7; 95% CI, 1.5-15.2; p = 0.01). Sensitivity analyses using Cox regression confirmed this finding for 1-year survival (adjusted hazard ratio, 5.6; 95% CI, 1.3-23; p = 0.048). CONCLUSION:The CT-defined hDAI was not an independent predictor of unfavorable short- and long-term outcomes and should not be used for acute prognostication in msTBI patients. Predominant hDAI patients had good clinical outcomes when supported to intensive care unit discharge and beyond. LEVEL OF EVIDENCE:Prognostic study, level III.

Dataset Information

Don't lose samples to estimation.

Publications

Don't lose samples to estimation.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets