Dataset Information

Cost-sensitive Performance Metric for Comparing Multiple Ordinal Classifiers.

ABSTRACT: The surge of interest in personalized and precision medicine during recent years has increased the application of ordinal classification problems in biomedical science. Currently, accuracy, Kendall's ?b , and average mean absolute error are three commonly used metrics for evaluating the effectiveness of an ordinal classifier. Although there are benefits to each, no single metric considers the benefits of predictive accuracy with the tradeoffs of misclassification cost. In addition, decision analysis that considers pairwise analysis of the metrics is not trivial due to inconsistent findings. A new cost-sensitive metric is proposed to find the optimal tradeoff between the two most critical performance measures of a classification task - accuracy and cost. The proposed method accounts for an inherent ordinal data structure, total misclassification cost of a classifier, and imbalanced class distribution. The strengths of the new methodology are demonstrated through analyses of three real cancer datasets and four simulation studies. The new cost-sensitive metric proved better performance in its ability to identify the best ordinal classifier for a given analysis. The performance metric devised in this study provides a comprehensive tool for comparative analysis of multiple (and competing) ordinal classifiers. Consideration of the tradeoff between accuracy and misclassification cost in decisions regarding ordinal classification problems is imperative in real-world application. The work presented here is a precursor to the possibility of incorporating the proposed metric into a prediction modeling algorithm for ordinal data as a means of integrating misclassification cost in final model selection.

SUBMITTER: George NI

PROVIDER: S-EPMC5217743 | biostudies-literature | 2016

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Cost-sensitive Performance Metric for Comparing Multiple Ordinal Classifiers.

George Nysia I NI Lu Tzu-Pin TP Chang Ching-Wei CW

Artificial intelligence research 20160115 1

The surge of interest in personalized and precision medicine during recent years has increased the application of ordinal classification problems in biomedical science. Currently, accuracy, Kendall's τb , and average mean absolute error are three commonly used metrics for evaluating the effectiveness of an ordinal classifier. Although there are benefits to each, no single metric considers the benefits of predictive accuracy with the tradeoffs of misclassification cost. I ...[more]

PMID: 28070226

Similar Datasets

Project description:Robust maximum likelihood (RML) and asymptotically generalized least squares (AGLS) methods have been recommended for fitting ordinal structural equation models. Studies show that some of these methods underestimate standard errors. However, these studies have not investigated the coverage and bias of interval estimates. An estimate with a reasonable standard error could still be severely biased. This can only be known by systematically investigating the interval estimates. The present study compares Bayesian, RML, and AGLS interval estimates of factor correlations in ordinal confirmatory factor analysis models (CFA) for small sample data. Six sample sizes, 3 factor correlations, and 2 factor score distributions (multivariate normal and multivariate mildly skewed) were studied. Two Bayesian prior specifications, informative and relatively less informative were studied. Undercoverage of confidence intervals and underestimation of standard errors was common in non-Bayesian methods. Underestimated standard errors may lead to inflated Type-I error rates. Non-Bayesian intervals were more positive biased than negatively biased, that is, most intervals that did not contain the true value were greater than the true value. Some non-Bayesian methods had non-converging and inadmissible solutions for small samples and non-normal data. Bayesian empirical standard error estimates for informative and relatively less informative priors were closer to the average standard errors of the estimates. The coverage of Bayesian credibility intervals was closer to what was expected with overcoverage in a few cases. Although some Bayesian credibility intervals were wider, they reflected the nature of statistical uncertainty that comes with the data (e.g., small sample). Bayesian point estimates were also more accurate than non-Bayesian estimates. The results illustrate the importance of analyzing coverage and bias of interval estimates, and how ignoring interval estimates can be misleading. Therefore, editors and policymakers should continue to emphasize the inclusion of interval estimates in research.

Project description:ImportanceUS hospitals report data on many health care quality metrics to government and independent health care rating organizations, but the annual cost to acute care hospitals of measuring and reporting quality metric data, independent of resources spent on quality interventions, is not well known.ObjectiveTo evaluate externally reported inpatient quality metrics for adult patients and estimate the cost of data collection and reporting, independent of quality-improvement efforts.Design, setting, and participantsRetrospective time-driven activity-based costing study at the Johns Hopkins Hospital (Baltimore, Maryland) with hospital personnel involved in quality metric reporting processes interviewed between January 1, 2019, and June 30, 2019, about quality reporting activities in the 2018 calendar year.Main outcomes and measuresOutcomes included the number of metrics, annual person-hours per metric type, and annual personnel cost per metric type.ResultsA total of 162 unique metrics were identified, of which 96 (59.3%) were claims-based, 107 (66.0%) were outcome metrics, and 101 (62.3%) were related to patient safety. Preparing and reporting data for these metrics required an estimated 108 478 person-hours, with an estimated personnel cost of $5 038 218.28 (2022 USD) plus an additional $602 730.66 in vendor fees. Claims-based (96 metrics; $37 553.58 per metric per year) and chart-abstracted (26 metrics; $33 871.30 per metric per year) metrics used the most resources per metric, while electronic metrics consumed far less (4 metrics; $1901.58 per metric per year).Conclusions and relevanceSignificant resources are expended exclusively for quality reporting, and some methods of quality assessment are far more expensive than others. Claims-based metrics were unexpectedly found to be the most resource intensive of all metric types. Policy makers should consider reducing the number of metrics and shifting to electronic metrics, when possible, to optimize resources spent in the overall pursuit of higher quality.

Project description:BACKGROUND:In stroke studies, ordinal logistic regression (OLR) is often used to analyze outcome on the modified Rankin Scale (mRS), whereas the non-parametric Mann-Whitney measure of superiority (MWS) has also been suggested. It is unclear how these perform comparatively when confounding adjustment is warranted. AIMS:Our aim is to quantify the performance of OLR and MWS in different confounding variable settings. METHODS:We set up a simulation study with three different scenarios; (1) dichotomous confounding variables, (2) continuous confounding variables, and (3) confounding variable settings mimicking a study on functional outcome after stroke. We compared adjusted ordinal logistic regression (aOLR) and stratified Mann-Whitney measure of superiority (sMWS), and also used propensity scores to stratify the MWS (psMWS). For comparability, OLR estimates were transformed to a MWS. We report bias, the percentage of runs that produced a point estimate deviating by more than 0.05 points (point estimate variation), and the coverage probability. RESULTS:In scenario 1, there was no bias in both sMWS and aOLR, with similar point estimate variation and coverage probabilities. In scenario 2, sMWS resulted in more bias (0.04 versus 0.00), and higher point estimate variation (41.6% versus 3.3%), whereas coverage probabilities were similar. In scenario 3, there was no bias in both methods, point estimate variation was higher in the sMWS (6.7%) versus aOLR (1.1%), and coverage probabilities were 0.98 (sMWS) versus 0.95 (aOLR). With psMWS, bias remained 0.00, with less point estimate variation (1.5%) and a coverage probability of 0.95. CONCLUSIONS:The bias of both adjustment methods was similar in our stroke simulation scenario, and the higher point estimate variation in the MWS improved with propensity score based stratification. The stratified MWS is a valid alternative for adjusted OLR only when the ratio of number of strata versus number of observations is relatively low, but propensity score based stratification extends the application range of the MWS.

Dataset Information

Cost-sensitive Performance Metric for Comparing Multiple Ordinal Classifiers.

Publications

Cost-sensitive Performance Metric for Comparing Multiple Ordinal Classifiers.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets