Dataset Information

The Net Reclassification Index (NRI): a Misleading Measure of Prediction Improvement Even with Independent Test Data Sets.

ABSTRACT: The Net Reclassification Index (NRI) is a very popular measure for evaluating the improvement in prediction performance gained by adding a marker to a set of baseline predictors. However, the statistical properties of this novel measure have not been explored in depth. We demonstrate the alarming result that the NRI statistic calculated on a large test dataset using risk models derived from a training set is likely to be positive even when the new marker has no predictive information. A related theoretical example is provided in which an incorrect risk function that includes an uninformative marker is proven to erroneously yield a positive NRI. Some insight into this phenomenon is provided. Since large values for the NRI statistic may simply be due to use of poorly fitting risk models, we suggest caution in using the NRI as the basis for marker evaluation. Other measures of prediction performance improvement, such as measures derived from the ROC curve, the net benefit function and the Brier score, cannot be large due to poorly fitting risk functions.

SUBMITTER: Pepe MS

PROVIDER: S-EPMC4615606 | biostudies-literature | 2015 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

The Net Reclassification Index (NRI): a Misleading Measure of Prediction Improvement Even with Independent Test Data Sets.

Pepe Margaret S MS Fan Jing J Feng Ziding Z Gerds Thomas T Hilden Jorgen J

Statistics in biosciences 20140823 2

The Net Reclassification Index (NRI) is a very popular measure for evaluating the improvement in prediction performance gained by adding a marker to a set of baseline predictors. However, the statistical properties of this novel measure have not been explored in depth. We demonstrate the alarming result that the NRI statistic calculated on a large test dataset using risk models derived from a training set is likely to be positive even when the new marker has no predictive information. A related ...[more]

PMID: 26504496

Similar Datasets

Project description:Net reclassification indices have recently become popular statistics for measuring the prediction increment of new biomarkers. We review the various types of net reclassification indices and their correct interpretations. We evaluate the advantages and disadvantages of quantifying the prediction increment with these indices. For predefined risk categories, we relate net reclassification indices to existing measures of the prediction increment. We also consider statistical methodology for constructing confidence intervals for net reclassification indices and evaluate the merits of hypothesis testing based on such indices. We recommend that investigators using net reclassification indices should report them separately for events (cases) and nonevents (controls). When there are two risk categories, the components of net reclassification indices are the same as the changes in the true- and false-positive rates. We advocate the use of true- and false-positive rates and suggest it is more useful for investigators to retain the existing, descriptive terms. When there are three or more risk categories, we recommend against net reclassification indices because they do not adequately account for clinically important differences in shifts among risk categories. The category-free net reclassification index is a new descriptive device designed to avoid predefined risk categories. However, it experiences many of the same problems as other measures such as the area under the receiver operating characteristic curve. In addition, the category-free index can mislead investigators by overstating the incremental value of a biomarker, even in independent validation data. When investigators want to test a null hypothesis of no prediction increment, the well-established tests for coefficients in the regression model are superior to the net reclassification index. If investigators want to use net reclassification indices, confidence intervals should be calculated using bootstrap methods rather than published variance formulas. The preferred single-number summary of the prediction increment is the improvement in net benefit.

Project description:BackgroundThe durability of Long Lasting Insecticidal Nets (LLINs) in field conditions is of great importance for malaria prevention and control efforts; however, the physical integrity of the net fabric is not well understood making it challenging to determine overall effectiveness of nets as they age. The 2011 World Health Organization Pesticide Evaluation Scheme (WHOPES) guidelines provide a simple, standardized method using a proportional hole index (PHI) for assessing net damage with the intent to provide national malaria control programs with guidelines to assess the useful life of LLINS and estimate the rate of replacement.MethodsWe evaluated the utility of the PHI measure using 409 LLINs collected over three years in Nampula Province, Mozambique following a mass distribution campaign in 2008. For each LLIN the diameter and distance from the bottom of the net were recorded for every hole. Holes were classified into four size categories and a PHI was calculated following WHOPES guidelines. We investigate how the size, shape, and location of holes influence the PHI. The areas of the WHOPES defined categories were compared to circular and elliptical areas based on approximate shape and actual measured axes of each hole and the PHI was compared to cumulative damaged surface area of the LLIN.ResultsThe damaged area of small, medium, large, and extra-large holes was overestimated using the WHOPES categories compared to elliptical areas using the actual measured axes. Similar results were found when comparing to circular areas except for extra-large holes which were underestimated. (Wilcoxon signed rank test of differences p< 0.0001 for all sizes). Approximating holes as circular overestimated hole surface area by 1.5 to 2 times or more. There was a significant difference in the mean number of holes < 0.5 cm by brand and there were more holes of all sizes on the bottom of nets than the top. For a range of hypothetical PHI thresholds used to designate a "failed LLIN", roughly 75 to 80% of failed LLINs were detected by considering large and extra-large holes alone, but sensitivity varied by brand.ConclusionsFuture studies may refine the PHI to better approximate overall damaged surface area. Furthermore, research is needed to identify whether or not appropriate PHI thresholds can be used to deem a net no longer protective. Once a cutoff is selected, simpler methods of determining the effective lifespan of LLINs can help guide replacement strategies for malaria control programs.

Dataset Information

The Net Reclassification Index (NRI): a Misleading Measure of Prediction Improvement Even with Independent Test Data Sets.

Publications

The Net Reclassification Index (NRI): a Misleading Measure of Prediction Improvement Even with Independent Test Data Sets.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets