Dataset Information

Net reclassification indices for evaluating risk prediction instruments: a critical review.

ABSTRACT: Net reclassification indices have recently become popular statistics for measuring the prediction increment of new biomarkers. We review the various types of net reclassification indices and their correct interpretations. We evaluate the advantages and disadvantages of quantifying the prediction increment with these indices. For predefined risk categories, we relate net reclassification indices to existing measures of the prediction increment. We also consider statistical methodology for constructing confidence intervals for net reclassification indices and evaluate the merits of hypothesis testing based on such indices. We recommend that investigators using net reclassification indices should report them separately for events (cases) and nonevents (controls). When there are two risk categories, the components of net reclassification indices are the same as the changes in the true- and false-positive rates. We advocate the use of true- and false-positive rates and suggest it is more useful for investigators to retain the existing, descriptive terms. When there are three or more risk categories, we recommend against net reclassification indices because they do not adequately account for clinically important differences in shifts among risk categories. The category-free net reclassification index is a new descriptive device designed to avoid predefined risk categories. However, it experiences many of the same problems as other measures such as the area under the receiver operating characteristic curve. In addition, the category-free index can mislead investigators by overstating the incremental value of a biomarker, even in independent validation data. When investigators want to test a null hypothesis of no prediction increment, the well-established tests for coefficients in the regression model are superior to the net reclassification index. If investigators want to use net reclassification indices, confidence intervals should be calculated using bootstrap methods rather than published variance formulas. The preferred single-number summary of the prediction increment is the improvement in net benefit.

SUBMITTER: Kerr KF

PROVIDER: S-EPMC3918180 | biostudies-literature | 2014 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Net reclassification indices for evaluating risk prediction instruments: a critical review.

Kerr Kathleen F KF Wang Zheyu Z Janes Holly H McClelland Robyn L RL Psaty Bruce M BM Pepe Margaret S MS

Epidemiology (Cambridge, Mass.) 20140101 1

Net reclassification indices have recently become popular statistics for measuring the prediction increment of new biomarkers. We review the various types of net reclassification indices and their correct interpretations. We evaluate the advantages and disadvantages of quantifying the prediction increment with these indices. For predefined risk categories, we relate net reclassification indices to existing measures of the prediction increment. We also consider statistical methodology for constru ...[more]

PMID: 24240655

Similar Datasets

Project description:ObjectiveTo systematically review how safety-net hospitals' status is identified and defined, discuss current definitions' limitations, and provide recommendations for a new classification and evaluation framework.Data sourcesSafety-net hospital-related studies in the MEDLINE database published before May 16, 2019.Study designSystematic review of the literature that adheres to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.Data collection/extraction methodsWe followed standard selection protocol, whereby studies went through an abstract review followed by a full-text screening for eligibility. For each included study, we extracted information about the identification method itself, including the operational definition, the dimension(s) of disadvantage reflected, study objective, and how safety-net status was evaluated.Principal findingsOur review identified 132 studies investigating safety-net hospitals. Analysis of identification methodologies revealed substantial heterogeneity in the ways disadvantage is defined, measured, and summarized at the hospital level, despite a 4.5-fold increase in studies investigating safety-net hospitals for the past decade. Definitions often exclusively used low-income proxies captured within existing health system data, rarely incorporated external social risk factor measures, and were commonly separated into distinct safety-net status categories when analyzed.ConclusionsConsistency in research and improvement in policy both require a standard definition for identifying safety-net hospitals. Yet no standardized definition of safety-net hospitals is endorsed and existing definitions have key limitations. Moving forward, approaches rooted in health equity theory can provide a more holistic framework for evaluating disadvantage at the hospital level. Furthermore, advancements in precision public health technologies make it easier to incorporate detailed neighborhood-level social determinants of health metrics into multidimensional definitions. Other countries, including the United Kingdom and New Zealand, have used similar methods of identifying social need to determine more accurate assessments of hospital performance and the development of policies and targeted programs for improving outcomes.

Project description:BACKGROUND:The ability to predict risk allows healthcare providers to propose which patients might benefit most from certain therapies, and is relevant to payers' demands to justify clinical and economic value. To understand the robustness of risk prediction models for heart failure (HF), we conducted a systematic literature review to (1) identify HF risk-prediction models, (2) assess statistical approach and extent of validation, (3) identify common variables, and (4) assess risk of bias (ROB). METHODS:Literature databases were searched from March 2013 to May 2018 to identify risk prediction models conducted in an out-of-hospital setting in adults with HF. Distinct risk prediction variables were ranked according to outcomes assessed and incorporation into the studies. ROB was assessed using Prediction model Risk Of Bias ASsessment Tool (PROBAST). RESULTS:Of 4720 non-duplicated citations, 40 risk-prediction publications were deemed relevant. Within the 40 publications, 58 models assessed 55 (co)primary outcomes, including all-cause mortality (n = 17), cardiovascular death (n = 9), HF hospitalizations (n = 15), and composite endpoints (n = 14). Few publications reported detail on handling missing data (n = 11; 28%). The discriminatory ability for predicting all-cause mortality, cardiovascular death, and composite endpoints was generally better than for HF hospitalization. 105 distinct predictor variables were identified. Predictors included in >5 publications were: N-terminal prohormone brain-natriuretic peptide, creatinine, blood urea nitrogen, systolic blood pressure, sodium, NYHA class, left ventricular ejection fraction, heart rate, and characteristics including male sex, diabetes, age, and BMI. Only 11/58 (19%) models had overall low ROB, based on our application of PROBAST. In total, 26/58 (45%) models discussed internal validation, and 14/58 (24%) external validation. CONCLUSIONS:The majority of the 58 identified risk-prediction models for HF present particular concerns according to ROB assessment, mainly due to lack of validation and calibration. The potential utility of novel approaches such as machine learning tools is yet to be determined. REGISTRATION NUMBER:The SLR was registered in Prospero (ID: CRD42018100709).

Project description:Multiple single biomarkers have been associated with poor outcomes in acute lung injury; however, no single biomarker has sufficient discriminating power to clearly indicate prognosis. Using both derivation and replication cohorts, we tested novel risk reclassification methods to determine whether measurement of multiple plasma biomarkers at the time of acute lung injury diagnosis would improve mortality prediction in acute lung injury.Analysis of plasma biomarker levels and prospectively collected clinical data from patients enrolled in two randomized controlled trials of ventilator therapy for acute lung injury.Intensive care units of university hospitals participating in the National Institutes of Health Acute Respiratory Distress Syndrome Network.Subjects enrolled in a trial of lower tidal volume ventilation (derivation cohort) and subjects enrolled in a trial of higher vs. lower positive end-expiratory pressure (replication cohort).None.The plasma biomarkers were intercellular adhesion molecule-1, von Willebrand factor, interleukin-8, soluble tumor necrosis factor receptor-1, and surfactant protein-D. In the derivation cohort (n = 547), adding data on these biomarkers to clinical predictors (Acute Physiology and Chronic Health Evaluation III score) at the time of study enrollment improved the accuracy of risk prediction, as reflected by a net reclassification improvement of 22% (95% confidence interval 13% to 32%; p < .001). In the replication cohort (n = 500), the net reclassification improvement was 17% (95% confidence interval 7% to 26%; p < .001). A reduced set of three biomarkers (interleukin-8, soluble tumor necrosis factor receptor-1, and surfactant protein-D) had nearly equivalent prognostic value in both cohorts.When combined with clinical data, plasma biomarkers measured at the onset of acute lung injury can improve the accuracy of risk prediction. Combining three or more biomarkers may be useful for selecting a high-risk acute lung injury population for enrollment in clinical trials of novel therapies.

Dataset Information

Net reclassification indices for evaluating risk prediction instruments: a critical review.

Publications

Net reclassification indices for evaluating risk prediction instruments: a critical review.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets