Dataset Information

Tournament leave-pair-out cross-validation for receiver operating characteristic analysis.

ABSTRACT: Receiver operating characteristic analysis is widely used for evaluating diagnostic systems. Recent studies have shown that estimating an area under receiver operating characteristic curve with standard cross-validation methods suffers from a large bias. The leave-pair-out cross-validation has been shown to correct this bias. However, while leave-pair-out produces an almost unbiased estimate of area under receiver operating characteristic curve, it does not provide a ranking of the data needed for plotting and analyzing the receiver operating characteristic curve. In this study, we propose a new method called tournament leave-pair-out cross-validation. This method extends leave-pair-out by creating a tournament from pair comparisons to produce a ranking for the data. Tournament leave-pair-out preserves the advantage of leave-pair-out for estimating area under receiver operating characteristic curve, while it also allows performing receiver operating characteristic analyses. We have shown using both synthetic and real-world data that tournament leave-pair-out is as reliable as leave-pair-out for area under receiver operating characteristic curve estimation and confirmed the bias in leave-one-out cross-validation on low-dimensional data. As a case study on receiver operating characteristic analysis, we also evaluate how reliably sensitivity and specificity can be estimated from tournament leave-pair-out receiver operating characteristic curves.

SUBMITTER: Montoya Perez I

PROVIDER: S-EPMC6745617 | biostudies-literature | 2019 Oct-Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Tournament leave-pair-out cross-validation for receiver operating characteristic analysis.

Montoya Perez Ileana I Airola Antti A Boström Peter J PJ Jambor Ivan I Pahikkala Tapio T

Statistical methods in medical research 20180820 10-11

Receiver operating characteristic analysis is widely used for evaluating diagnostic systems. Recent studies have shown that estimating an area under receiver operating characteristic curve with standard cross-validation methods suffers from a large bias. The leave-pair-out cross-validation has been shown to correct this bias. However, while leave-pair-out produces an almost unbiased estimate of area under receiver operating characteristic curve, it does not provide a ranking of the data needed f ...[more]

PMID: 30126322

Similar Datasets

Project description:BACKGROUND:Despite some advances, cardiovascular disease (CVD) remains the leading cause of death and healthcare costs in the United States. We therefore developed a comprehensive CVD policy simulation model that identifies cost-effective approaches for reducing CVD burden. This paper aims to: 1) describe our model in detail; and 2) perform model validation analyses. METHODS:The model simulates 1,000,000 adults (ages 35 to 80 years) using a variety of CVD-related epidemiological data, including previously calibrated Framingham-based risk scores for coronary heart disease and stroke. We validated our microsimulation model using recent National Health and Nutrition Examination Survey (NHANES) data, with baseline values collected in 1999-2000 and cause-specific mortality follow-up through 2011. Model-based (simulated) results were compared to observed all-cause and CVD-specific mortality data (from NHANES) for the same starting population using survival curves and, in a method not typically used for disease model validation, receiver operating characteristic (ROC) curves. RESULTS:Observed 10-year all-cause mortality in NHANES v. the simulation model was 11.2% (95% CI, 10.3% to 12.2%) v. 10.9%; corresponding results for CVD mortality were 2.2% (1.8% to 2.7%) v. 2.6%. Areas under the ROC curves for model-predicted 10-year all-cause and CVD mortality risks were 0.83 (0.81 to 0.85) and 0.84 (0.81 to 0.88), respectively; corresponding results for 5-year risks were 0.80 (0.77 to 0.83) and 0.81 (0.75 to 0.87), respectively. LIMITATIONS:The model is limited by the uncertainties in the data used to estimate its input parameters. Additionally, our validation analyses did not include non-fatal CVD outcomes due to NHANES data limitations. CONCLUSIONS:The simulation model performed well in matching to observed nationally representative longitudinal mortality data. ROC curve analysis, which has been traditionally used for risk prediction models, can also be used to assess discrimination for disease simulation models.

Dataset Information

Tournament leave-pair-out cross-validation for receiver operating characteristic analysis.

Publications

Tournament leave-pair-out cross-validation for receiver operating characteristic analysis.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets