Making External Validation Valid for Molecular Classifier Development.
Ontology highlight
ABSTRACT: Purpose
Accurate assessment of a molecular classifier that guides patient care is of paramount importance in precision oncology. Recent years have seen an increasing use of external validation for such assessment. However, little is known about how it is affected by ubiquitous unwanted variations in test data because of disparate experimental handling and by the use of data normalization for alleviating such variations.Methods
In this paper, we studied these issues using two microarray data sets for the same set of tumor samples and additional data simulated by resampling under various levels of signal-to-noise ratio and different designs for array-to-sample allocation.Results
We showed that (1) unwanted variations can lead to biased classifier assessment and (2) data normalization mitigates the bias to varying extents depending on the specific method used. In particular, frozen normalization methods for test data outperform their conventional forms in terms of both reducing the bias in accuracy estimation and increasing robustness to handling effects. We make available our benchmarking tool as an R package on GitHub for performing such evaluation on additional methods for normalization and classification.Conclusion
Our findings thus highlight the importance of proper test-data normalization for valid assessment by external validation and call for caution on the choice of normalization method for molecular classifier development.
SUBMITTER: Wu Y
PROVIDER: S-EPMC8345919 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Background Opioid misuse screening in hospitals is resource-intensive and rarely done. Many hospitalized patients are never offered opioid treatment. An automated approach leveraging routinely captured electronic health record (EHR) data may be easier for hospitals to institute. We previously derived and internally validated an opioid classifier in a separate hospital setting. The aim is to externally validate our previously published and open-source machine-learning classifier at a different hospital for identifying cases of opioid misuse. Methods An observational cohort of 56,227 adult hospitalizations was examined between October 2017 and December 2019 during a hospital-wide substance use screening program with manual screening. Manually completed Drug Abuse Screening Test served as the reference standard to validate a convolutional neural network (CNN) classifier with coded word embedding features from the clinical notes of the EHR. The opioid classifier utilized all notes in the EHR and sensitivity analysis was also performed on the first 24 h of notes. Calibration was performed to account for the lower prevalence than in the original cohort. Results Manual screening for substance misuse was completed in 67.8% (n?=?56,227) with 1.1% (n?=?628) identified with opioid misuse. The data for external validation included 2,482,900 notes with 67,969 unique clinical concept features. The opioid classifier had an AUC of 0.99 (95% CI 0.99–0.99) across the encounter and 0.98 (95% CI 0.98–0.99) using only the first 24 h of notes. In the calibrated classifier, the sensitivity and positive predictive value were 0.81 (95% CI 0.77–0.84) and 0.72 (95% CI 0.68–0.75). For the first 24 h, they were 0.75 (95% CI 0.71–0.78) and 0.61 (95% CI 0.57–0.64). Conclusions Our opioid misuse classifier had good discrimination during external validation. Our model may provide a comprehensive and automated approach to opioid misuse identification that augments current workflows and overcomes manual screening barriers. Supplementary Information The online version contains supplementary material available at 10.1186/s13722-021-00229-7.
| S-EPMC7967783 | biostudies-literature
Project description:Purpose: We sought to develop and evaluate a diagnostic classifier of UC subtype with the goal of accurate classification from clinically available specimens. Methods: Tumor samples from 52 patients with high-grade UC were profiled for BASE47 genes concurrently by RNAseq as well as NanoString. After design and technical validation of a BASE47 NanoString probeset, results from the RNAseq and NanoString were used to translate diagnostic criteria to the Nanostring platform. Evaluation of repeatability and accuracy was performed to derive a final Nanostring based classifier. Diagnostic classification resulting from the NanoString BASE47 classifier was validated on an independent dataset (n=63). The training and validation datasets accurately classified 87% and 93% of samples, respectively. Results: We have derived a NanoString-platform BASE47 classifier that accurately predicts basal-like and luminal-like subtypes in high grade urothelial cancer. We have further validated our new NanoString BASE47 classifier on an independent dataset and confirmed high accuracy when compared with our original Transcriptome BASE47 classifier. Conclusions: The NanoString BASE47 classifier provides a faster turnaround time, a lower cost per sample to process, and maintains the accuracy of the original subtype classifier for better clinical implementation.
2020-11-03 | GSE160693 | GEO
Project description:BackgroundRecent molecular characterization of urothelial cancer (UC) has suggested potential pathways in which to direct treatment, leading to a host of targeted therapies in development for UC. In parallel, gene expression profiling has demonstrated that high-grade UC is a heterogeneous disease. Prognostic basal-like and luminal-like subtypes have been identified and an accurate transcriptome BASE47 classifier has been developed. However, these phenotypes cannot be broadly investigated due to the lack of a clinically viable diagnostic assay. We sought to develop and evaluate a diagnostic classifier of UC subtype with the goal of accurate classification from clinically available specimens.MethodsTumor samples from 52 patients with high-grade UC were profiled for BASE47 genes concurrently by RNAseq as well as NanoString. After design and technical validation of a BASE47 NanoString probeset, results from the RNAseq and NanoString were used to translate diagnostic criteria to the Nanostring platform. Evaluation of repeatability and accuracy was performed to derive a final Nanostring based classifier. Diagnostic classification resulting from the NanoString BASE47 classifier was validated on an independent dataset (n = 30). The training and validation datasets accurately classified 87% and 93% of samples, respectively.ResultsHere we have derived a NanoString-platform BASE47 classifier that accurately predicts basal-like and luminal-like subtypes in high grade urothelial cancer. We have further validated our new NanoString BASE47 classifier on an independent dataset and confirmed high accuracy when compared with our original Transcriptome BASE47 classifier.ConclusionsThe NanoString BASE47 classifier provides a faster turnaround time, a lower cost per sample to process, and maintains the accuracy of the original subtype classifier for better clinical implementation.
| S-EPMC7745986 | biostudies-literature
Project description:Background: Necroptosis, a novel form of apoptosis, plays a crucial function in the progression of colon adenocarcinoma (COAD) and is expected to be triggered in cancer therapy for enhancing anti-tumor immunity. However, the function of necroptosis in tumors and its relationship with the tumor microenvironment (TME) remains largely unclear. Methods: Necroptosis-related genes (NRGs) were collected from high-quality literature. Using The Cancer Genome Atlas (TCGA) (https://cancergenome.nih.gov) and the Gene Expression Omnibus (GEO) (www.ncbi.nlm.nih.gov/geo) meta-cohorts, a robust risk model was constructed to systematically examine the clinical value, functional status, the role of TME based on the risk model, as also the genomic variations. Results: A risk model containing nine NRGs, including TNF receptor-associated factor (TRAF2), TNF receptor 1 associated via death domain (TRADD), ubiquitin carboxyl-terminal hydrolase 21 (USP21), TNF receptor superfamily, member 6 (FAS), tumor necrosis factor receptor superfamily 10B (TNFRSF10B), mitogen-activated protein kinase 8 (MAPK8), mixed lineage kinase domain-like (MLKL), TNF receptor-associated factor 5 (TRAF5), and recombinant receptor-interacting serine-threonine kinase 3 (RIPK3), was constructed. The risk model's stability and accuracy were demonstrated in training, as also the validation cohorts; it was verified as an independent prognostic model for COAD. High-risk group patients developed "cold" tumors having active tumor proliferation and immunosuppression, while those in the low-risk group developed "hot" tumors with active immune and cell killing functions. Moreover, a higher number of copy number variations in the genome and fewer somatic mutations were found in high-risk group patients. Furthermore, higher sensitivity towards immunotherapy and chemotherapy was seen in patients of the low-risk group. Conclusion: A reliable risk model based on NRGs to assess patient prognosis and guide clinical decision-making was constructed and validated. Our findings may contribute to the understanding of necroptosis and are expected to aid clinical management and guide precision treatment for patients with COAD.
| S-EPMC9389450 | biostudies-literature
Project description:PurposeConsensus molecular subtyping (CMS) of colorectal cancer has potential to reshape the colorectal cancer landscape. We developed and validated an assay that is applicable on formalin-fixed, paraffin-embedded (FFPE) samples of colorectal cancer and implemented the assay in a Clinical Laboratory Improvement Amendments (CLIA)-certified laboratory.Experimental designWe performed an in silico experiment to build an optimal CMS classifier using a training set of 1,329 samples from 12 studies and validation set of 1,329 samples from 14 studies. We constructed an assay on the basis of NanoString CodeSets for the top 472 genes, and performed analyses on paired flash-frozen (FF)/FFPE samples from 175 colorectal cancers to adapt the classifier to FFPE samples using a subset of genes found to be concordant between FF and FFPE, tested the classifier's reproducibility and repeatability, and validated in a CLIA-certified laboratory. We assessed prognostic significance of CMS in 345 patients pooled across three clinical trials.ResultsThe best classifier was weighted support vector machine with high accuracy across platforms and gene lists (>0.95), and the 472-gene model outperforming existing classifiers. We constructed subsets of 99 and 200 genes with high FF/FFPE concordance, and adapted FFPE-based classifier that had strong classification accuracy (>80%) relative to "gold standard" CMS. The classifier was reproducible to sample type and RNA quality, and demonstrated poor prognosis for CMS1-3 and good prognosis for CMS2 in metastatic colorectal cancer (P < 0.001).ConclusionsWe developed and validated a colorectal cancer CMS assay that is ready for use in clinical trials, to assess prognosis in standard-of-care settings and explore as predictor of therapy response.
| S-EPMC8713413 | biostudies-literature
Project description:Background Immunotherapy and sorafenib exert anti-tumor effects via ferroptosis, but reliable biomarkers for the individual treatment and prognosis prediction of hepatocellular carcinoma (HCC) based on the ferroptosis and immune status remain lacking. Methods Ferroptosis-related genes (FRGs) were identified by downloading data from FerrDb and by searching and reading original articles from PubMed. Immune-related genes (IRGs) were downloaded from ImmPort. Prognostic FRGs and IRGs in the GSE14520 (n = 220) and The Cancer Genome Atlas (TCGA, n = 365) datasets were identified. Least absolute shrinkage and selection operator (LASSO) Cox regression and multivariate Cox regression were used for model construction. Ferroptosis expression profiles, the infiltration of immune cells, and the somatic mutation status were analyzed and compared. Results Twenty-seven prognostic ferroptosis- and immune-related signatures were included to construct a comprehensive index of ferroptosis and immune status (CIFI). A subgroup of patients was identified as having a high CIFI value, which was associated with a worse prognosis. This subgroup of patients had significantly up-regulated expressions of many suppressors of ferroptosis and higher fractions of immunosuppressive cells, such as cancer-associated fibroblasts (CAFs) and myeloid-derived suppressor cells (MDSCs). Notably, somatic mutation analysis indicated that high-CIFI patients had higher levels of tumor heterogeneity and higher mutation frequencies of genes like TP53. Conclusion In this work, a novel prognostic classifier was developed based on ferroptosis- and IRGs in HCC, and this classifier could be used for prognostic prediction and the selection of patients for immunotherapies and targeted therapies.
| S-EPMC7785857 | biostudies-literature
Project description:Lung cancer is the leading cause of cancer-related deaths worldwide. Hypoxia is a crucial microenvironmental factor in lung adenocarcinoma (LUAD). However, the prognostic value based on hypoxia and immune in LUAD remains to be further clarified. The hypoxia-related genes (HRGs) and immune-related genes (IRGs) were downloaded from the public database. The RNA-seq expression and matched complete clinical data for LUAD were retrieved from The Cancer Genome Atlas (TCGA) and Gene Expression Omnibus (GEO) database. The least absolute shrinkage and selection operator (LASSO) Cox regression analysis was applied to model construction. Hypoxia expression profiles, immune cell infiltration, functional enrichment analysis, Tumor Immune Dysfunction and Exclusion (TIDE) score and the somatic mutation status were analyzed and compared based on the model. Moreover, immunofluorescence (IF) staining in human LUAD cases to explore the expression of hypoxia marker and immune checkpoint. A prognostic model of 9 genes was established, which can divide patients into two subgroups. There were obvious differences in hypoxia and immune characteristics in the two groups, the group with high-risk score value showed significantly high expression of hypoxia genes and programmed death ligand-1 (PD-L1), and maybe more sensitive to immunotherapy. Patients in the high-risk group had shorter overall survival (OS). This model has a good predictive value for the prognosis of LUAD. We constructed a new HRGs and IRGs model for prognostic prediction of LUAD. This model may benefit future immunotherapy for LUAD.
| S-EPMC9174870 | biostudies-literature
Project description:BackgroundApproximately 50% of patients with uveal melanoma (UM) will develop metastatic disease, usually involving the liver. The outcome of metastatic UM (mUM) is generally poor and no standard therapy has been established. Additionally, clinicians lack a validated prognostic tool to evaluate these patients. The aim of this work was to develop a reliable prognostic nomogram for clinicians.Patients and methodsTwo cohorts of mUM patients, from Veneto Oncology Institute (IOV) (N=152) and Mayo Clinic (MC) (N=102), were analyzed to develop and externally validate, a prognostic nomogram.ResultsThe median survival of mUM was 17.2 months in the IOV cohort and 19.7 in the MC cohort. Percentage of liver involvement (HR 1.6), elevated levels of serum LDH (HR 1.6), and a WHO performance status=1 (HR 1.5) or 2-3 (HR 4.6) were associated with worse prognosis. Longer disease-free interval from diagnosis of UM to that of mUM conferred a survival advantage (HR 0.9). The nomogram had a concordance probability of 0.75 (SE .006) in the development dataset (IOV), and 0.80 (SE .009) in the external validation (MC). Nomogram predictions were well calibrated.ConclusionsThe nomogram, which includes percentage of liver involvement, LDH levels, WHO performance status and disease free-interval accurately predicts the prognosis of mUM and could be useful for decision-making and risk stratification for clinical trials.
| S-EPMC4363319 | biostudies-literature
Project description:Organ shortage leads to using non-optimal liver grafts. Thus, to determine the graft quality, the Donor Risk Index and the Eurotransplant Donor Risk Index have been proposed. In a previous study we showed that neither could be validated on the French database. Our aim was then dedicated to propose an adaptive Donor Quality Index (DQI) using data from 3961 liver transplantation (LT) performed in France between 2009 and 2013, with an external validation based on 1048 French LT performed in 2014. Using Cox models and three different methods of selection, we developed a new score and defined groups at risk. Model performance was assessed by means of three measures of discrimination corrected by the optimism using a bootstrap procedure. An external validation was also performed in order to evaluate its calibration and discrimination. Five donor covariates were retained: age, cause of death, intensive care unit stay, lowest MDRD creatinine clearance, and liver type. Three groups at risk could be discriminated. The performances of the model were satisfactory after internal validation. Calibration and discrimination were preserved in the external validation dataset. The DQI exhibited good properties and is potentially adaptive as an aid for better guiding decision making for LT.
| S-EPMC6026153 | biostudies-literature
Project description:HIF-1 (hypoxia-inducible factor 1) signaling played a vital role in HCC (hepatocellular carcinoma) prognosis. We aimed to establish an accurate risk scoring system for HCC prognosis prediction and treatment guidance. 424 samples from TCGA (The Cancer Genome Atlas) and 445 samples from GSE14520 dataset were included as the derivation and validation cohort, respectively. In the derivation cohort, prognostic relevant signatures were selected from sixteen HIF-1 related genes and LASSO regression was adopted for model construction. Tumor-infiltrating immune cells were calculated using CIBERSORT algorithm. HIF-1 signaling significantly increased in HCC samples compared with normal tissues. Scoring system based on SLC2A1, ENO1, LDHA and GAPDH exhibited a continuous predictive ability for OS (overall survival) in HCC patients. PCA and t-SNE analysis confirmed a reliable clustering ability of risk score in both cohorts. Patients were classified into high-risk and low-risk groups and the survival outcomes between the two groups showed significant differences. In the derivation cohort, Cox regression indicated the scoring system was an independent predictor for OS, which was validated in the validation cohort. Different infiltrating immune cells fraction and immune scores were also observed in different groups. Herein, a novel integrated scoring system was developed based on HIF-1 related genes, which would be conducive to the precise treatment of patients.
| S-EPMC7066907 | biostudies-literature