ROC and AUC with a Binary Predictor: a Potentially Misleading Metric.
Ontology highlight
ABSTRACT:
SUBMITTER: Muschelli J
PROVIDER: S-EPMC7695228 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BackgroundThe area under the receiver operating characteristic (ROC) curve, referred to as the AUC, is an appropriate measure for describing the overall accuracy of a diagnostic test or a biomarker in early phase trials without having to choose a threshold. There are many approaches for estimating the confidence interval for the AUC. However, all are relatively complicated to implement. Furthermore, many approaches perform poorly for large AUC values or small sample sizes.MethodsThe AUC is actually a probability. So we propose a modified Wald interval for a single proportion, which can be calculated on a pocket calculator. We performed a simulation study to compare this modified Wald interval (without and with continuity correction) with other intervals regarding coverage probability and statistical power.ResultsThe main result is that the proposed modified Wald intervals maintain and exploit the type I error much better than the intervals of Agresti-Coull, Wilson, and Clopper-Pearson. The interval suggested by Bamber, the Mann-Whitney interval without transformation and also the interval of the binormal AUC are very liberal. For small sample sizes the Wald interval with continuity has a comparable coverage probability as the LT interval and higher power. For large sample sizes the results of the LT interval and of the Wald interval without continuity correction are comparable.ConclusionsIf individual patient data is not available, but only the estimated AUC and the total sample size, the modified Wald intervals can be recommended as confidence intervals for the AUC. For small sample sizes the continuity correction should be used.
| S-EPMC3938139 | biostudies-literature
Project description:BackgroundSome patients administered cholesterol-lowering therapies may experience an increase in the proportion of small LDL particles, which may be misinterpreted as a worsening of atherosclerotic coronary heart disease risk. This study assessed the lipid effects of adding ezetimibe to atorvastatin or doubling the atorvastatin dose on low-density lipoprotein cholesterol (LDL-C) levels (and the cholesterol content of LDL subclasses), LDL particle number (approximated by apolipoprotein B), and LDL particle size. This was a multicenter, double-blind, randomized, parallel-group study of hypercholesterolemic, high atherosclerotic coronary heart disease risk patients. After stabilization of atorvastatin 40 mg, 579 patients with LDL-C >70 mg/dL were randomized to 6 weeks of ezetimibe + atorvastatin 40 mg or atorvastatin 80 mg. Efficacy parameters included changes from baseline in LDL-C, apolipoprotein B, non-high-density lipoprotein cholesterol (non-HDL-C), and lipoprotein subclasses (Vertical Auto Profile II) and pattern for the overall population, as well as patient subgroups with baseline triglyceride levels <150 mg/dL or ≥150 mg/dL.ResultsBoth treatments significantly reduced LDL-C (and the cholesterol content of most LDL subfractions [LDL1-4]) apolipoprotein B, non-HDL-C levels, but did not reduce the proportion of smaller, more dense LDL particles; in fact, the proportion of Pattern B was numerically increased. Results were generally similar in patients with triglyceride levels <150 or ≥150 mg/dL.ConclusionsWhen assessing the effects of escalating cholesterol-lowering therapy, effects upon Pattern B alone to assess coronary heart disease risk may be misleading when interpreted without considerations of other lipid effects, such as reductions in LDL-C, atherogenic lipoprotein particle concentration, and non-HDL-C levels.Trial registration(Registered at clinicaltrials.gov: Clinical trial # NCT00276484).
| S-EPMC3012666 | biostudies-literature
Project description:This study aimed to evaluate a quality metric that identifies pediatric potentially avoidable transfers from diagnosis and procedure codes. Using physician medical record review as the gold standard, the following steps were used: (1) develop the initial metric definition, (2) estimate initial metric definition operating characteristics, (3) refine this definition to optimize the c-statistic, and (4) validate this optimized metric definition using a separate sample. The initial metric using Sample A patient transfers had a c-statistic of 0.63 (95% confidence interval = 0.53-0.73). Following 22 revisions, the optimized metric definition was a transfer discharged within 24 hours that did not receive any of a select list of 60 268 specialized diagnoses or procedures. The optimized metric on Sample B demonstrated a sensitivity of 80.6%, specificity of 85.7%, and c-statistic of 0.83 (95% confidence interval = 0.75-0.91). The quality metric developed and validated in this study demonstrated satisfactory operating characteristics, providing a feasible means to measure this important outcome.
| S-EPMC6901803 | biostudies-literature
Project description:When comparing binary test results from two diagnostic systems, superiority in both "sensitivity" and "specificity" also implies differences in all conventional summary indices and locally in the underlying receiver operating characteristics (ROC) curves. However, when one of the two binary tests has higher sensitivity and lower specificity (or vice versa), comparisons of their performance levels are nontrivial and the use of different summary indices may lead to contradictory conclusions. A frequently used approach that is free of subjectivity associated with summary indices is based on the comparison of the underlying ROC curves that requires the collection of rating data using multicategory scales, whether natural or experimentally imposed. However, data for reliable estimation of ROC curves are frequently unavailable. The purpose of this article is to develop an approach of using "diagnostic likelihood ratios", namely, likelihood ratios of "positive" or "negative" responses, to make simple inferences regarding the underlying ROC curves and associated areas in the absence of reliable rating data or regarding the relative binary characteristics, when these are of primary interest.For inferences related to underlying curves, the authors exploit the assumption of concavity of the true underlying ROC curve to describe conditions under which these curves have to be different and under which the curves have different areas. For scenarios when the binary characteristics are of primary interest, the authors use characteristics of "chance performance" to demonstrate that the derived conditions provide strong evidence of superiority of one binary test as compared to another. By relating these derived conditions to hypotheses about the true likelihood ratios of two binary diagnostic tests being compared, the authors enable a straightforward statistical procedure for the corresponding inferences.The authors derived simple algebraic and graphical methods for describing the conditions for superiority of one of two diagnostic tests with respect to their binary characteristics, the underlying ROC curves, or the areas under the curves. The graphical regions are useful for identifying potential differences between two systems, which then have to be tested statistically. The simple statistical tests can be performed with well known methods for comparison of diagnostic likelihood ratios. The developed approach offers a solution for some of the more difficult to analyze scenarios, where diagnostic tests do not demonstrate concordant differences in terms of both sensitivity and specificity. In addition, the resulting inferences do not contradict the conclusions that can be obtained using conventional and reasonably defined summary indices.When binary diagnostic tests are of primary interest, the proposed approach offers an objective and powerful method for comparing two binary diagnostic tests. The significant advantage of this method is that it enables objective analyses when one test has higher sensitivity but lower specificity, while ensuring agreement with study conclusions based on other reasonable and widely acceptable summary indices. For truly multicategory diagnostic tests, the proposed method can help in concluding inferiority of one of the diagnostic tests based on binary data, thereby potentially saving the need for conducting a more expensive multicategory ROC study.
| S-EPMC2980542 | biostudies-other
Project description:Online videos have become a prevalent means for people to acquire information. Videos, however, are often polarized, misleading, or contain topics on which people have different, contradictory views. In this work, we introduce natural language explanations to stimulate more deliberate reasoning about videos and raise users' awareness of potentially deceiving or biased information. With these explanations, we aim to support users in actively deciding and reflecting on the usefulness of the videos. We generate the explanations through an end-to-end pipeline that extracts reflection triggers so users receive additional information to the video based on its source, covered topics, communicated emotions, and sentiment. In a between-subjects user study, we examine the effect of showing the explanations for videos on three controversial topics. Besides, we assess the users' alignment with the video's message and how strong their belief is about the topic. Our results indicate that respondents' alignment with the video's message is critical to evaluate the video's usefulness. Overall, the explanations were found to be useful and of high quality. While the explanations do not influence the perceived usefulness of the videos compared to only seeing the video, people with an extreme negative alignment with a video's message perceived it as less useful (with or without explanations) and felt more confident in their assessment. We relate our findings to cognitive dissonance since users seem to be less receptive to explanations when the video's message strongly challenges their beliefs. Given these findings, we provide a set of design implications for explanations grounded in theories on reducing cognitive dissonance in light of raising awareness about online deception.
| S-EPMC8507585 | biostudies-literature
Project description:Binary classifiers are routinely evaluated with performance measures such as sensitivity and specificity, and performance is frequently illustrated with Receiver Operating Characteristics (ROC) plots. Alternative measures such as positive predictive value (PPV) and the associated Precision/Recall (PRC) plots are used less frequently. Many bioinformatics studies develop and evaluate classifiers that are to be applied to strongly imbalanced datasets in which the number of negatives outweighs the number of positives significantly. While ROC plots are visually appealing and provide an overview of a classifier's performance across a wide range of specificities, one can ask whether ROC plots could be misleading when applied in imbalanced classification scenarios. We show here that the visual interpretability of ROC plots in the context of imbalanced datasets can be deceptive with respect to conclusions about the reliability of classification performance, owing to an intuitive but wrong interpretation of specificity. PRC plots, on the other hand, can provide the viewer with an accurate prediction of future classification performance due to the fact that they evaluate the fraction of true positives among positive predictions. Our findings have potential implications for the interpretation of a large number of studies that use ROC plots on imbalanced datasets.
| S-EPMC4349800 | biostudies-literature
Project description:The aim of this in vivo study was to compare the efficacy of vancomycin at standard doses (VAN-SD) to that of VAN at adjusted doses (VAN-AD) in achieving a VAN area under the curve/MIC ratio (AUC/MIC) of ?400 against three methicillin-resistant Staphylococcus aureus (MRSA) strains with different microdilution VAN MICs in an experimental endocarditis model. The valve vegetation bacterial counts after 48 h of VAN therapy were compared, and no differences were observed between the two treatment groups for any of the three strains tested. Overall, for VAN-SD and VAN-AD, the rates of sterile vegetations were 15/45 (33.3%) and 21/49 (42.8%) (P = 0.343), while the medians (interquartile ranges [IQRs]) for log10 CFU/g of vegetation were 2 (0 to 6.9) and 2 (0 to 4.5) (P = 0.384), respectively. In conclusion, this VAN AUC/MIC pharmacodynamic target was not a good predictor of vancomycin efficacy in MRSA experimental endocarditis.
| S-EPMC5444163 | biostudies-literature
Project description:Progress in analytical ultracentrifugation (AUC) has been hindered by obstructions to hardware innovation and by software incompatibility. In this paper, we announce and outline the Open AUC Project. The goals of the Open AUC Project are to stimulate AUC innovation by improving instrumentation, detectors, acquisition and analysis software, and collaborative tools. These improvements are needed for the next generation of AUC-based research. The Open AUC Project combines on-going work from several different groups. A new base instrument is described, one that is designed from the ground up to be an analytical ultracentrifuge. This machine offers an open architecture, hardware standards, and application programming interfaces for detector developers. All software will use the GNU Public License to assure that intellectual property is available in open source format. The Open AUC strategy facilitates collaborations, encourages sharing, and eliminates the chronic impediments that have plagued AUC innovation for the last 20 years. This ultracentrifuge will be equipped with multiple and interchangeable optical tracks so that state-of-the-art electronics and improved detectors will be available for a variety of optical systems. The instrument will be complemented by a new rotor, enhanced data acquisition and analysis software, as well as collaboration software. Described here are the instrument, the modular software components, and a standardized database that will encourage and ease integration of data analysis and interpretation software.
| S-EPMC2812709 | biostudies-literature
Project description:Longitudinal data with binary repeated responses are now widespread among clinical studies and standard statistical analysis methods have become inadequate in the answering of clinical hypotheses. Instead of such conventional approaches, statisticians have started proposing better techniques, such as the Generalized Estimating Equations (GEE) approach and Generalized Linear Mixed Models (GLMM) technique. In this research, we undertook a comparative study of modeling binary repeated responses using an anesthesiology dataset which has 375 patient data with clinical variables. We modeled the relationship between hypotension and age, gender, surgical department, positions of patients during surgery, diastolic blood pressure, pulse, electrocardiography and doses of Marcain-heavy, chirocaine, fentanyl, and midazolam. Moreover, parameter estimates between the GEE and the GLMM were compared. The parameter estimates, except time-after, Marcain-Heavy, and Fentanyl from the GLMM, are larger than those from GEE. The standard errors from the GLMM are larger than those from GEE. GLMM appears to be more suitable approach than the GEE approach for the analysis hypotension during spinal anesthesia.
| S-EPMC4217193 | biostudies-other
Project description:Atherosclerotic plaque rupture in carotid arteries can lead to stroke which is one of the leading causes of death or disability worldwide. The accumulation of atherosclerotic plaque in an artery changes the mechanical properties of the vessel. Whilst healthy arteries can continuously adapt to mechanical loads by remodelling their internal structure, particularly the load-bearing collagen fibres, diseased vessels may have limited remodelling capabilities. In this study, a local stress modulated remodelling algorithm is proposed to explore the mechanical response of arterial tissue to the remodelling of collagen fibres. This stress driven remodelling algorithm is used to predict the optimum distribution of fibres in healthy and diseased human carotid bifurcations obtained using Magnetic Resonance Imaging (MRI). In the models, healthy geometries were segmented into two layers: media and adventitia and diseased into four components: adventitia, media, plaque atheroma and lipid pool (when present in the MRI images). A novel meshing technique for hexahedral meshing of these geometries is also demonstrated. Using the remodelling algorithm, the optimum fibre patterns in various patient specific plaques are identified and the role that deviations from these fibre configurations in plaque vulnerability is shown. This study provides critical insights into the collagen fibre patterns required in carotid artery and plaque tissue to maintain plaque stability.
| S-EPMC8586512 | biostudies-literature