Visualizing Validation of Protein Surface Classifiers.
Ontology highlight
ABSTRACT:
SUBMITTER: Sarikaya A
PROVIDER: S-EPMC4204728 | biostudies-literature | 2014 Jun
REPOSITORIES: biostudies-literature
Publications
Visualizing Validation of Protein Surface Classifiers.
Sarikaya A A Albers D D Mitchell J J Gleicher M M
Computer graphics forum : journal of the European Association for Computer Graphics 20140601 3
Many bioinformatics applications construct classifiers that are validated in experiments that compare their results to known ground truth over a corpus. In this paper, we introduce an approach for exploring the results of such classifier validation experiments, focusing on classifiers for regions of molecular surfaces. We provide a tool that allows for examining classification performance patterns over a test corpus. The approach combines a summary view that provides information about an entire ...[more]
Similar Datasets
Project description:Proposed molecular classifiers may be overfit to idiosyncrasies of noisy genomic and proteomic data. Cross-validation methods are often used to obtain estimates of classification accuracy, but both simulations and case studies suggest that, when inappropriate methods are used, bias may ensue. Bias can be bypassed and generalizability can be tested by external (independent) validation. We evaluated 35 studies that have reported on external validation of a molecular classifier. We extracted information on study design and methodological features, and compared the performance of molecular classifiers in internal cross-validation versus external validation for 28 studies where both had been performed. We demonstrate that the majority of studies pursued cross-validation practices that are likely to overestimate classifier performance. Most studies were markedly underpowered to detect a 20% decrease in sensitivity or specificity between internal cross-validation and external validation [median power was 36% (IQR, 21-61%) and 29% (IQR, 15-65%), respectively]. The median reported classification performance for sensitivity and specificity was 94% and 98%, respectively, in cross-validation and 88% and 81% for independent validation. The relative diagnostic odds ratio was 3.26 (95% CI 2.04-5.21) for cross-validation versus independent validation. Finally, we reviewed all studies (n = 758) which cited those in our study sample, and identified only one instance of additional subsequent independent validation of these classifiers. In conclusion, these results document that many cross-validation practices employed in the literature are potentially biased and genuine progress in this field will require adoption of routine external validation of molecular classifiers, preferably in much larger studies than in current practice.
| S-EPMC3088312 | biostudies-literature
Project description:We formulate the estimation of monotone response surface of multiple factors as the inverse of an iteration of partially ordered classifier ensembles. Each ensemble (called PIPE-classifiers) is a projection of Bayes classifiers on the constrained space. We prove the inverse of PIPE-classifiers (iPIPE) exists, and propose algorithms to efficiently compute iPIPE by reducing the space over which optimisation is conducted. The methods are applied in analysis and simulation settings where the surface dimension is higher than what the isotonic regression literature typically considers. Simulation shows iPIPE-based credible intervals achieve nominal coverage probability and are more precise compared to unconstrained estimation.
| S-EPMC10919322 | biostudies-literature
Project description:In previous years a substantial number of studies have identified statistically important predictors of nursing home admission (NHA). However, as far as we know, the analyses have been done at the population-level. No prior research has analysed the prediction accuracy of a NHA model for individuals.This study is an analysis of 3056 longer-term home care customers in the city of Tampere, Finland. Data were collected from the records of social and health service usage and RAI-HC (Resident Assessment Instrument - Home Care) assessment system during January 2011 and September 2015. The aim was to find out the most efficient variable subsets to predict NHA for individuals and validate the accuracy. The variable subsets of predicting NHA were searched by sequential forward selection (SFS) method, a variable ranking metric and the classifiers of logistic regression (LR), support vector machine (SVM) and Gaussian naive Bayes (GNB). The validation of the results was guaranteed using randomly balanced data sets and cross-validation. The primary performance metrics for the classifiers were the prediction accuracy and AUC (average area under the curve).The LR and GNB classifiers achieved 78% accuracy for predicting NHA. The most important variables were RAI MAPLE (Method for Assigning Priority Levels), functional impairment (RAI IADL, Activities of Daily Living), cognitive impairment (RAI CPS, Cognitive Performance Scale), memory disorders (diagnoses G30-G32 and F00-F03) and the use of community-based health-service and prior hospital use (emergency visits and periods of care).The accuracy of the classifier for individuals was high enough to convince the officials of the city of Tampere to integrate the predictive model based on the findings of this study as a part of home care information system. Further work need to be done to evaluate variables that are modifiable and responsive to interventions.
| S-EPMC5390435 | biostudies-other
Project description:Determining protein-protein interaction (PPI) in biological systems is of considerable importance, and prediction of PPI has become a popular research area. Although different classifiers have been developed for PPI prediction, no single classifier seems to be able to predict PPI with high confidence. We postulated that by combining individual classifiers the accuracy of PPI prediction could be improved. We developed a method called protein-protein interaction prediction classifiers merger (PPCM), and this method combines output from two PPI prediction tools, GO2PPI and Phyloprof, using Random Forests algorithm. The performance of PPCM was tested by area under the curve (AUC) using an assembled Gold Standard database that contains both positive and negative PPI pairs. Our AUC test showed that PPCM significantly improved the PPI prediction accuracy over the corresponding individual classifiers. We found that additional classifiers incorporated into PPCM could lead to further improvement in the PPI prediction accuracy. Furthermore, cross species PPCM could achieve competitive and even better prediction accuracy compared to the single species PPCM. This study established a robust pipeline for PPI prediction by integrating multiple classifiers using Random Forests algorithm. This pipeline will be useful for predicting PPI in nonmodel species.
| S-EPMC4619929 | biostudies-literature
Project description:Challenges in determining the structures of heterogeneous and dynamic protein complexes have greatly hampered past efforts to obtain a mechanistic understanding of many important biological processes. One such process is chaperone-assisted protein folding. Obtaining structural ensembles of chaperone-substrate complexes would ultimately reveal how chaperones help proteins fold into their native state. To address this problem, we devised a new structural biology approach based on X-ray crystallography, termed residual electron and anomalous density (READ). READ enabled us to visualize even sparsely populated conformations of the substrate protein immunity protein 7 (Im7) in complex with the Escherichia coli chaperone Spy, and to capture a series of snapshots depicting the various folding states of Im7 bound to Spy. The ensemble shows that Spy-associated Im7 samples conformations ranging from unfolded to partially folded to native-like states and reveals how a substrate can explore its folding landscape while being bound to a chaperone.
| S-EPMC4937829 | biostudies-literature
Project description:Background: Current standard methods used to detect and monitor bladder cancer (BC) are invasive or have low sensitivity. We have previously reported in an international European study four non-invasive tests for BC diagnosis based on the gene expression patterns of urine. Objective: to validate the tests in an independent Asian cohort. Design, setting and participants: Prospective blinded study in which consecutive voided urine samples from BC patients and controls (n=520) were collected in the Fudan University Shanghai Cancer Center from 2014-2016. Gene expression values were quantified using TaqMan Arrays. The same cut-off as previously reported for discrimination between tumours and controls was used in this validation study. Results and limitations: Finally, a total of 257 tumour and 132 control urine samples were analysed. We found a high accuracy for the four gene classifiers in this independent Asian set, the classifiers composed of 5 and 10 genes achieved the best sensitivity (80.54% and 81.32%, respectively) maintaining a high specificity (91.67% and 85.61%, respectively). Sensitivity of 5-gene (GS_D5) and 10-gene (GS_D10) expression classifiers in recurrent BC cases (78 and 79%, respectively) is comparable to that of primary BC cases (82%). Cytology and NMP22 identified 67% and 40%, respectively, of tumours that have been diagnosed with our tests. In addition, influence of each studied gene was analyzed and showed similar gene rank between Chinese and Caucasian population. Conclusions: Our study proves that our non-invasive diagnostic BC tests can be reproduced in independent cohorts and in an external laboratory. All the four gene classifiers have shown equal or superior performance to the current gold standard in the present and previously reported validation studies. Consequently, they may be taken for consideration as molecular tests applicable to clinical practice in the management of BC. Patient summary: Our gene classifiers achieve sensitivities up to 90% in HR NMIBC and MIBC patients, while this achievement is comparatively lower in LR NMIBC ones.
| S-EPMC6134826 | biostudies-other
Project description:Background: Development and application of transcriptomics-based gene classifiers for ecotoxicological applications lag far behind those of human biomedical science. Many such classifiers discovered thus far lack vigorous statistical and experimental validations, with their stability and reliability unknown. A combination of genetic algorithm/support vector machines and genetic algorithm/K nearest neighbors were used in this study to search for classifiers of endocrine disrupting chemicals (EDCs) in zebrafish. Searches were conducted on both tissue-specific and all tissue combined datasets, either across the entire transcriptome or within individual transcription factor (TF) networks previously linked to EDC effects. Candidate classifiers were evaluated by gene set enrichment analysis (GSEA) on both the original training data and a dedicated validation dataset. Results: Multi-tissue dataset yielded no classifiers. Among the 19 chemical-tissue conditions evaluated, the transcriptome-wide searches yielded classifiers for six of them, each having approximately 20 to 30 gene features unique to a condition. Searches within individual TF networks produced classifiers for 15 chemical-tissue conditions, each containing 100 or fewer top-ranked gene features pooled from those of multiple TF networks and also unique to each condition. For the training dataset, 10 out of 11 classifiers successfully identified the gene expression profiles (GEPs) of their intended chemical-tissue conditions by GSEA. For the validation dataset, classifiers for prochloraz-ovary and flutamide-ovary also correctly identified the GEPs of corresponding conditions while no classifier could predict the GEP from prochloraz-brain. Conclusions: The discrepancies in the performance of these classifiers were attributed in part to varying data complexity among the conditions, as measured to some degree by Fisher’s discriminant ratio statistic. This variation in data complexity could likely be compensated by adjusting sample size for individual chemical-tissue conditions, thus suggesting a need for a preliminary survey of transcriptomic responses before launching a full scale classifier discovery effort. While GSEA appeared to provide a flexible and effective tool for application of gene classifiers, a similar but more refined algorithm, connectivity mapping, should also be explored for ecotoxicological applications. The distribution characteristics of classifiers across tissues, chemicals, and TF networks suggested a differential biological impact among the EDCs on zebrafish transcriptome involving some basic cellular functions. chemical abbreviations: EE2, 17α-ethynyl estradiol; FAD, fadrozole; TRB, 17 -trenbolone; FIP, fipronil; PRO, prochloraz; FLU, flutamide; MUS, muscimol; KET, ketoconazole; TRI, trilostane; VIN, vinclozolin
2012-08-22 | GSE38070 | GEO
Project description:Important three-dimensional spatial features such as depth and surface concavity can be difficult to convey clearly in the context of two-dimensional images. In the area of macromolecular visualization, the computer graphics technique of ray-tracing can be helpful, but further techniques for emphasizing surface concavity can give clearer perceptions of depth. The notion of diffusion accessibility is well-suited for emphasizing such features of macromolecular surfaces, but a method for calculating diffusion accessibility has not been made widely available. Here we make available a web-based platform that performs the necessary calculation by solving the Laplace equation for steady state diffusion, and produces scripts for visualization that emphasize surface depth by coloring according to diffusion accessibility. The URL is http://services.mbi.ucla.edu/DiffAcc/.
| S-EPMC4594669 | biostudies-literature
Project description:Background: Development and application of transcriptomics-based gene classifiers for ecotoxicological applications lag far behind those of human biomedical science. Many such classifiers discovered thus far lack vigorous statistical and experimental validations, with their stability and reliability unknown. A combination of genetic algorithm/support vector machines and genetic algorithm/K nearest neighbors were used in this study to search for classifiers of endocrine disrupting chemicals (EDCs) in zebrafish. Searches were conducted on both tissue-specific and all tissue combined datasets, either across the entire transcriptome or within individual transcription factor (TF) networks previously linked to EDC effects. Candidate classifiers were evaluated by gene set enrichment analysis (GSEA) on both the original training data and a dedicated validation dataset. Results: Multi-tissue dataset yielded no classifiers. Among the 19 chemical-tissue conditions evaluated, the transcriptome-wide searches yielded classifiers for six of them, each having approximately 20 to 30 gene features unique to a condition. Searches within individual TF networks produced classifiers for 15 chemical-tissue conditions, each containing 100 or fewer top-ranked gene features pooled from those of multiple TF networks and also unique to each condition. For the training dataset, 10 out of 11 classifiers successfully identified the gene expression profiles (GEPs) of their intended chemical-tissue conditions by GSEA. For the validation dataset, classifiers for prochloraz-ovary and flutamide-ovary also correctly identified the GEPs of corresponding conditions while no classifier could predict the GEP from prochloraz-brain. Conclusions: The discrepancies in the performance of these classifiers were attributed in part to varying data complexity among the conditions, as measured to some degree by Fisher’s discriminant ratio statistic. This variation in data complexity could likely be compensated by adjusting sample size for individual chemical-tissue conditions, thus suggesting a need for a preliminary survey of transcriptomic responses before launching a full scale classifier discovery effort. While GSEA appeared to provide a flexible and effective tool for application of gene classifiers, a similar but more refined algorithm, connectivity mapping, should also be explored for ecotoxicological applications. The distribution characteristics of classifiers across tissues, chemicals, and TF networks suggested a differential biological impact among the EDCs on zebrafish transcriptome involving some basic cellular functions. chemical abbreviations: EE2, 17α-ethynyl estradiol; FAD, fadrozole; TRB, 17 -trenbolone; FIP, fipronil; PRO, prochloraz; FLU, flutamide; MUS, muscimol; KET, ketoconazole; TRI, trilostane; VIN, vinclozolin Since this study was conducted in several phases, three different version of Agilent zebrafish two color microarrays were used based on their availability at the time. These include G2518A (designID 013223) and G2519F (designID 015064, 019161). There were a total of 58 treatment conditions with various combinations of chemical, tissue type, exposure time, and gender. Each condition contained eight to 12 independent samples, half from chemical-treated fish and half from water- control fish.
2012-08-22 | E-GEOD-38070 | biostudies-arrayexpress
Project description:BACKGROUND: Protein complexes are basic cellular entities that carry out the functions of their components. It can be found that in databases of protein complexes of yeast like CYC2008, the major type of known protein complexes is heterodimeric complexes. Although a number of methods for trying to predict sets of proteins that form arbitrary types of protein complexes simultaneously have been proposed, it can be found that they often fail to predict heterodimeric complexes. RESULTS: In this paper, we have designed several features characterizing heterodimeric protein complexes based on genomic data sets, and proposed a supervised-learning method for the prediction of heterodimeric protein complexes. This method learns the parameters of the features, which are embedded in the naïve Bayes classifier. The log-likelihood ratio derived from the naïve Bayes classifier with the parameter values obtained by maximum likelihood estimation gives the score of a given pair of proteins to predict whether the pair is a heterodimeric complex or not. A five-fold cross-validation shows good performance on yeast. The trained classifiers also show higher predictability than various existing algorithms on yeast data sets with approximate and exact matching criteria. CONCLUSIONS: Heterodimeric protein complex prediction is a rather harder problem than heteromeric protein complex prediction because heterodimeric protein complex is topologically simpler. However, it turns out that by designing features specialized for heterodimeric protein complexes, predictability of them can be improved. Thus, the design of more sophisticate features for heterodimeric protein complexes as well as the accumulation of more accurate and useful genome-wide data sets will lead to higher predictability of heterodimeric protein complexes. Our tool can be downloaded from http://imi.kyushu-u.ac.jp/~om/.
| S-EPMC4219333 | biostudies-other