Dataset Information

Making External Validation Valid for Molecular Classifier Development.

ABSTRACT:

Purpose

Accurate assessment of a molecular classifier that guides patient care is of paramount importance in precision oncology. Recent years have seen an increasing use of external validation for such assessment. However, little is known about how it is affected by ubiquitous unwanted variations in test data because of disparate experimental handling and by the use of data normalization for alleviating such variations.

Methods

In this paper, we studied these issues using two microarray data sets for the same set of tumor samples and additional data simulated by resampling under various levels of signal-to-noise ratio and different designs for array-to-sample allocation.

Results

We showed that (1) unwanted variations can lead to biased classifier assessment and (2) data normalization mitigates the bias to varying extents depending on the specific method used. In particular, frozen normalization methods for test data outperform their conventional forms in terms of both reducing the bias in accuracy estimation and increasing robustness to handling effects. We make available our benchmarking tool as an R package on GitHub for performing such evaluation on additional methods for normalization and classification.

Conclusion

Our findings thus highlight the importance of proper test-data normalization for valid assessment by external validation and call for caution on the choice of normalization method for molecular classifier development.

SUBMITTER: Wu Y

PROVIDER: S-EPMC8345919 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:Background Opioid misuse screening in hospitals is resource-intensive and rarely done. Many hospitalized patients are never offered opioid treatment. An automated approach leveraging routinely captured electronic health record (EHR) data may be easier for hospitals to institute. We previously derived and internally validated an opioid classifier in a separate hospital setting. The aim is to externally validate our previously published and open-source machine-learning classifier at a different hospital for identifying cases of opioid misuse. Methods An observational cohort of 56,227 adult hospitalizations was examined between October 2017 and December 2019 during a hospital-wide substance use screening program with manual screening. Manually completed Drug Abuse Screening Test served as the reference standard to validate a convolutional neural network (CNN) classifier with coded word embedding features from the clinical notes of the EHR. The opioid classifier utilized all notes in the EHR and sensitivity analysis was also performed on the first 24 h of notes. Calibration was performed to account for the lower prevalence than in the original cohort. Results Manual screening for substance misuse was completed in 67.8% (n?=?56,227) with 1.1% (n?=?628) identified with opioid misuse. The data for external validation included 2,482,900 notes with 67,969 unique clinical concept features. The opioid classifier had an AUC of 0.99 (95% CI 0.99–0.99) across the encounter and 0.98 (95% CI 0.98–0.99) using only the first 24 h of notes. In the calibrated classifier, the sensitivity and positive predictive value were 0.81 (95% CI 0.77–0.84) and 0.72 (95% CI 0.68–0.75). For the first 24 h, they were 0.75 (95% CI 0.71–0.78) and 0.61 (95% CI 0.57–0.64). Conclusions Our opioid misuse classifier had good discrimination during external validation. Our model may provide a comprehensive and automated approach to opioid misuse identification that augments current workflows and overcomes manual screening barriers. Supplementary Information The online version contains supplementary material available at 10.1186/s13722-021-00229-7.

Project description:Background and aimsUnhealthy alcohol use (UAU) is one of the leading causes of global morbidity. A machine learning approach to alcohol screening could accelerate best practices when integrated into electronic health record (EHR) systems. This study aimed to validate externally a natural language processing (NLP) classifier developed at an independent medical center.DesignRetrospective cohort study.SettingThe site for validation was a midwestern United States tertiary-care, urban medical center that has an inpatient structured universal screening model for unhealthy substance use and an active addiction consult service.Participants/casesUnplanned admissions of adult patients between October 23, 2017 and December 31, 2019, with EHR documentation of manual alcohol screening were included in the cohort (n = 57 605).MeasurementsThe Alcohol Use Disorders Identification Test (AUDIT) served as the reference standard. AUDIT scores ≥5 for females and ≥8 for males served as cases for UAU. To examine error in manual screening or under-reporting, a post hoc error analysis was conducted, reviewing discordance between the NLP classifier and AUDIT-derived reference. All clinical notes excluding the manual screening and AUDIT documentation from the EHR were included in the NLP analysis.FindingsUsing clinical notes from the first 24 hours of each encounter, the NLP classifier demonstrated an area under the receiver operating characteristic curve (AUCROC) and precision-recall area under the curve (PRAUC) of 0.91 (95% CI = 0.89-0.92) and 0.56 (95% CI = 0.53-0.60), respectively. At the optimal cut point of 0.5, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were 0.66 (95% CI = 0.62-0.69), 0.98 (95% CI = 0.98-0.98), 0.35 (95% CI = 0.33-0.38), and 1.0 (95% CI = 1.0-1.0), respectively.ConclusionsExternal validation of a publicly available alcohol misuse classifier demonstrates adequate sensitivity and specificity for routine clinical use as an automated screening tool for identifying at-risk patients.

Project description:Background: Necroptosis, a novel form of apoptosis, plays a crucial function in the progression of colon adenocarcinoma (COAD) and is expected to be triggered in cancer therapy for enhancing anti-tumor immunity. However, the function of necroptosis in tumors and its relationship with the tumor microenvironment (TME) remains largely unclear. Methods: Necroptosis-related genes (NRGs) were collected from high-quality literature. Using The Cancer Genome Atlas (TCGA) (https://cancergenome.nih.gov) and the Gene Expression Omnibus (GEO) (www.ncbi.nlm.nih.gov/geo) meta-cohorts, a robust risk model was constructed to systematically examine the clinical value, functional status, the role of TME based on the risk model, as also the genomic variations. Results: A risk model containing nine NRGs, including TNF receptor-associated factor (TRAF2), TNF receptor 1 associated via death domain (TRADD), ubiquitin carboxyl-terminal hydrolase 21 (USP21), TNF receptor superfamily, member 6 (FAS), tumor necrosis factor receptor superfamily 10B (TNFRSF10B), mitogen-activated protein kinase 8 (MAPK8), mixed lineage kinase domain-like (MLKL), TNF receptor-associated factor 5 (TRAF5), and recombinant receptor-interacting serine-threonine kinase 3 (RIPK3), was constructed. The risk model's stability and accuracy were demonstrated in training, as also the validation cohorts; it was verified as an independent prognostic model for COAD. High-risk group patients developed "cold" tumors having active tumor proliferation and immunosuppression, while those in the low-risk group developed "hot" tumors with active immune and cell killing functions. Moreover, a higher number of copy number variations in the genome and fewer somatic mutations were found in high-risk group patients. Furthermore, higher sensitivity towards immunotherapy and chemotherapy was seen in patients of the low-risk group. Conclusion: A reliable risk model based on NRGs to assess patient prognosis and guide clinical decision-making was constructed and validated. Our findings may contribute to the understanding of necroptosis and are expected to aid clinical management and guide precision treatment for patients with COAD.

Project description:BackgroundThe Sepsis-3 criteria introduced the system that uses the Sequential Organ-Failure Assessment (SOFA) score to define sepsis. The cardiovascular SOFA (CV SOFA) scoring system needs modification due to the change in guideline-recommended vasopressors. In this study, we aimed to develop and to validate the modified CV SOFA score.MethodsWe developed, internally validated, and externally validated the modified CV SOFA score using the suspected infection cohort, sepsis cohort, and septic shock cohort. The primary outcome was 28-day mortality. The modified CV SOFA score system was constructed with consideration of the recently recommended use of the vasopressor norepinephrine with or without lactate level. The predictive validity of the modified SOFA score was evaluated by the discrimination for the primary outcome. Discrimination was assessed using the area under the receiver operating characteristics curve (AUC). Calibration was assessed using the calibration curve. We compared the prognostic performance of the original CV/total SOFA score and the modified CV/total SOFA score to detect mortality in patients with suspected infection, sepsis, or septic shock.ResultsWe identified 7,393 patients in the suspected cohort, 4038 patients in the sepsis cohort, and 3,107 patients in the septic shock cohort in seven Korean emergency departments (EDs). The 28-day mortality rates were 7.9%, 21.4%, and 20.5%, respectively, in the suspected infection, sepsis, and septic shock cohorts. The model performance is higher when vasopressor and lactate were used in combination than the vasopressor only used model. The modified CV/total SOFA score was well-developed and internally and externally validated in terms of discrimination and calibration. Predictive validity of the modified CV SOFA was significantly higher than that of the original CV SOFA in the development set (0.682 vs 0.624, p < 0.001), test set (0.716 vs 0.638), and all other cohorts (0.648 vs 0.557, 0.674 vs 0.589). Calibration was modest. In the suspected infection cohort, the modified model classified more patients to sepsis (66.0 vs 62.5%) and identified more patients at risk of septic mortality than the SOFA score (92.6 vs 89.5%).ConclusionsAmong ED patients with suspected infection, sepsis, and septic shock, the newly-developed modified CV/total SOFA score had higher predictive validity and identified more patients at risk of septic mortality.