Classifier uncertainty: evidence, potential impact, and probabilistic treatment.
Ontology highlight
ABSTRACT:
SUBMITTER: Totsch N
PROVIDER: S-EPMC7959610 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BACKGROUND: An important step in strain optimization is to identify reactions whose activities should be modified to achieve the desired cellular objective. Preferably, these reactions are identified systematically, as the number of possible combinations of reaction modifications could be very large. Over the last several years, a number of computational methods have been described for identifying combinations of reaction modifications. However, none of these methods explicitly address uncertainties in implementing the reaction activity modifications. In this work, we model the uncertainties as probability distributions in the flux carrying capacities of reactions. Based on this model, we develop an optimization method that identifies reactions for flux capacity modifications to predict outcomes with high statistical likelihood. RESULTS: We compare three optimization methods that select an intervention set comprising up- or down-regulation of reaction flux capacity: CCOpt (Chance constrained optimization), DetOpt (Deterministic optimization), and MCOpt (Monte Carlo-based optimization). We evaluate the methods using a Monte Carlo simulation-based method, MCEval (Monte Carlo Evaluations). We present two case studies analyzing a CHO cell and an adipocyte model. The flux capacity distributions required for our methods were estimated from maximal reaction velocities or elementary mode analysis. The intervention set selected by CCOpt consistently outperforms the intervention set selected by DetOpt in terms of tolerance to flux capacity variations. MCEval shows that the optimal flux predicted based on the CCOpt intervention set is more likely to be obtained, in a probabilistic sense, than the flux predicted by DetOpt. The intervention sets identified by CCOpt and MCOpt were similar; however, the exhaustive sampling required by MCOpt incurred significantly greater computational cost. CONCLUSIONS: Maximizing tolerance to variable engineering outcomes (in modifying enzyme activities) can identify intervention sets that statistically improve the desired cellular objective.
| S-EPMC3626866 | biostudies-other
Project description:BackgroundOlfactory receptors (ORs), the largest mammalian gene superfamily (900-1400 genes), has >50% pseudogenes in humans. While most of these inactive genes are identified via coding frame (nonsense) disruptions, seemingly intact genes may also be inactive due to other deleterious (missense) mutations. An ultimate assessment of the actual size of the functional human OR repertoire thus requires an accurate distinction between genes and pseudogenes.ResultsTo characterize inactive ORs with intact open reading frame, we have developed a probabilistic Classifier for Olfactory Receptor Pseudogenes (CORP). This algorithm is based on deviations from a functionally crucial consensus, constituting sixty highly conserved positions identified by a comparison of two evolutionarily-constrained OR repertoires (mouse and dog) with a small pseudogene fraction. We used a logistic regression analysis to assign appropriate coefficients to the conserved position and thus achieving maximal separation between active and inactive ORs. Consequently, the algorithms identified only 5% of the mouse functional ORs as pseudogenes, setting an upper limit of 0.05 to the false positive detection. Finally we used this algorithm to classify the 384 purportedly intact human OR genes. Of these, 135 were predicted as likely encoding non-functional proteins, and 38 were segregating between active and inactive forms due to missense polymorphisms.ConclusionWe demonstrated that the CORP algorithm is capable to distinguish between functional and non-functional OR genes with high precision even when the encoded protein would differ by a single amino acid. Using the CORP algorithm, we predict that approximately 70% of human OR genes are likely non-functional pseudogenes, a much higher number than hitherto suspected. The method we present may be employed for better annotation of inactive members in other gene families as well. CORP algorithm is available at: http://bioportal.weizmann.ac.il/HORDE/CORP/
| S-EPMC1599758 | biostudies-literature
Project description:We produce probabilistic projections of population for all countries based on probabilistic projections of fertility, mortality, and migration. We compare our projections to those from the United Nations' Probabilistic Population Projections, which uses similar methods for fertility and mortality but deterministic migration projections. We find that uncertainty in migration projection is a substantial contributor to uncertainty in population projections for many countries. Prediction intervals for the populations of Northern America and Europe are over 70% wider, whereas prediction intervals for the populations of Africa, Asia, and the world as a whole are nearly unchanged. Out-of-sample validation shows that the model is reasonably well calibrated.
| S-EPMC4988608 | biostudies-literature
Project description:Electrophysiological recordings from freely behaving animals are a widespread and powerful mode of investigation in sleep research. These recordings generate large amounts of data that require sleep stage annotation (polysomnography), in which the data is parcellated according to three vigilance states: awake, rapid eye movement (REM) sleep, and non-REM (NREM) sleep. Manual and current computational annotation methods ignore intermediate states because the classification features become ambiguous, even though intermediate states contain important information regarding vigilance state dynamics. To address this problem, we have developed "Somnotate"-a probabilistic classifier based on a combination of linear discriminant analysis (LDA) with a hidden Markov model (HMM). First we demonstrate that Somnotate sets new standards in polysomnography, exhibiting annotation accuracies that exceed human experts on mouse electrophysiological data, remarkable robustness to errors in the training data, compatibility with different recording configurations, and an ability to maintain high accuracy during experimental interventions. However, the key feature of Somnotate is that it quantifies and reports the certainty of its annotations. We leverage this feature to reveal that many intermediate vigilance states cluster around state transitions, whereas others correspond to failed attempts to transition. This enables us to show for the first time that the success rates of different types of transition are differentially affected by experimental manipulations and can explain previously observed sleep patterns. Somnotate is open-source and has the potential to both facilitate the study of sleep stage transitions and offer new insights into the mechanisms underlying sleep-wake dynamics.
| S-EPMC10824458 | biostudies-literature
Project description:We develop measures of oil earnings uncertainty (OEU) using analyst forecasts drawn from a large firm-level dataset. OEU is related to future economic downturns, so some OEU measures may serve to forecast future downturns. An increase in OEU also has adverse effects on the US oil sector. The results are robust to conditioning on aggregate uncertainty. At the same time, OEU is related to increases in stock prices - unlike aggregate uncertainty, which has the opposite effect. OEU is thus an independent influence on both the oil industry and on economic aggregates.
| S-EPMC7320262 | biostudies-literature
Project description:Genetic sequencing is subject to many different types of errors, but most analyses treat the resultant sequences as if they are known without error. Next generation sequencing methods rely on significantly larger numbers of reads than previous sequencing methods in exchange for a loss of accuracy in each individual read. Still, the coverage of such machines is imperfect and leaves uncertainty in many of the base calls. In this work, we demonstrate that the uncertainty in sequencing techniques will affect downstream analysis and propose a straightforward method to propagate the uncertainty. Our method (which we have dubbed Sequence Uncertainty Propagation, or SUP) uses a probabilistic matrix representation of individual sequences which incorporates base quality scores as a measure of uncertainty that naturally lead to resampling and replication as a framework for uncertainty propagation. With the matrix representation, resampling possible base calls according to quality scores provides a bootstrap- or prior distribution-like first step towards genetic analysis. Analyses based on these re-sampled sequences will include a more complete evaluation of the error involved in such analyses. We demonstrate our resampling method on SARS-CoV-2 data. The resampling procedures add a linear computational cost to the analyses, but the large impact on the variance in downstream estimates makes it clear that ignoring this uncertainty may lead to overly confident conclusions. We show that SARS-CoV-2 lineage designations via Pangolin are much less certain than the bootstrap support reported by Pangolin would imply and the clock rate estimates for SARS-CoV-2 are much more variable than reported.
| S-EPMC10124968 | biostudies-literature
Project description:A tailor-made survey documents consumers' perceptions of the US economy's response to a large shock: the advent of the COVID-19 pandemic. The survey ran at a daily frequency between March 2020 and July 2021. Consumer's perceptions regarding output and inflation react rapidly. Uncertainty is pervasive. A business-cycle model calibrated to the consumers' views provides an interpretation. The rise in household uncertainty accounts for two-thirds of the fall in output. Different perceptions about monetary policy can explain why consumers and professional forecasters agree on the recessionary impact, but have sharply divergent views about inflation.
| S-EPMC8827064 | biostudies-literature
Project description:The concept of probabilistic parameter threshold analysis has recently been introduced as a way of probabilistic sensitivity analysis for decision-making under uncertainty, in particular, for health economic evaluations which compare two or more alternative treatments with consideration of uncertainty on outcomes and costs. In this paper we formulate the probabilistic threshold analysis as a root-finding problem involving the conditional expectations, and propose a pairwise stochastic approximation algorithm to search for the threshold value below and above which the choice of conditionally optimal decision options changes. Numerical experiments for both a simple synthetic testcase and a chemotherapy Markov model illustrate the effectiveness of our proposed algorithm, without any need for accurate estimation or approximation of conditional expectations which the existing approaches rely upon. Moreover we introduce a new measure called decision switching probability for probabilistic sensitivity analysis in this paper.
| S-EPMC8490445 | biostudies-literature
Project description:Machine learning is increasingly applied to Earth Observation (EO) data to obtain datasets that contribute towards international accords. However, these datasets contain inherent uncertainty that needs to be quantified reliably to avoid negative consequences. In response to the increased need to report uncertainty, we bring attention to the promise of conformal prediction within the domain of EO. Unlike previous uncertainty quantification methods, conformal prediction offers statistically valid prediction regions while concurrently supporting any machine learning model and data distribution. To support the need for conformal prediction, we reviewed EO datasets and found that only 22.5% of the datasets incorporated a degree of uncertainty information, with unreliable methods prevalent. Current open implementations require moving large amounts of EO data to the algorithms. We introduced Google Earth Engine native modules that bring conformal prediction to the data and compute, facilitating the integration of uncertainty quantification into existing traditional and deep learning modelling workflows. To demonstrate the versatility and scalability of these tools we apply them to valued EO applications spanning local to global extents, regression, and classification tasks. Subsequently, we discuss the opportunities arising from the use of conformal prediction in EO. We anticipate that accessible and easy-to-use tools, such as those provided here, will drive wider adoption of rigorous uncertainty quantification in EO, thereby enhancing the reliability of downstream uses such as operational monitoring and decision-making.
| S-EPMC11246475 | biostudies-literature
Project description:Predicting whether a lung nodule will grow, remain stable or regress over time, especially early in its follow-up, would help doctors prescribe personalized treatments and better surgical planning. However, the multifactorial nature of lung tumour progression hampers the identification of growth patterns. In this work, we propose a deep hierarchical generative and probabilistic network that, given an initial image of the nodule, predicts whether it will grow, quantifies its future size and provides its expected semantic appearance at a future time. Unlike previous solutions, our approach also estimates the uncertainty in the predictions from the intrinsic noise in medical images and the inter-observer variability in the annotations. The evaluation of this method on an independent test set reported a future tumour growth size mean absolute error of 1.74 mm, a nodule segmentation Dice's coefficient of 78% and a tumour growth accuracy of 84% on predictions made up to 24 months ahead. Due to the lack of similar methods for providing future lung tumour growth predictions, along with their associated uncertainty, we adapted equivalent deterministic and alternative generative networks (i.e., probabilistic U-Net, Bayesian test dropout and Pix2Pix). Our method outperformed all these methods, corroborating the adequacy of our approach.
| S-EPMC9689366 | biostudies-literature