Dataset Information

Added predictive value of omics data: specific issues related to validation illustrated by two case studies.

ABSTRACT: In the last years, the importance of independent validation of the prediction ability of a new gene signature has been largely recognized. Recently, with the development of gene signatures which integrate rather than replace the clinical predictors in the prediction rule, the focus has been moved to the validation of the added predictive value of a gene signature, i.e. to the verification that the inclusion of the new gene signature in a prediction model is able to improve its prediction ability.The high-dimensional nature of the data from which a new signature is derived raises challenging issues and necessitates the modification of classical methods to adapt them to this framework. Here we show how to validate the added predictive value of a signature derived from high-dimensional data and critically discuss the impact of the choice of methods on the results.The analysis of the added predictive value of two gene signatures developed in two recent studies on the survival of leukemia patients allows us to illustrate and empirically compare different validation techniques in the high-dimensional framework.The issues related to the high-dimensional nature of the omics predictors space affect the validation process. An analysis procedure based on repeated cross-validation is suggested.

SUBMITTER: De Bin R

PROVIDER: S-EPMC4271356 | biostudies-literature | 2014

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Added predictive value of omics data: specific issues related to validation illustrated by two case studies.

De Bin Riccardo R Herold Tobias T Boulesteix Anne-Laure AL

BMC medical research methodology 20141028

<h4>Background</h4>In the last years, the importance of independent validation of the prediction ability of a new gene signature has been largely recognized. Recently, with the development of gene signatures which integrate rather than replace the clinical predictors in the prediction rule, the focus has been moved to the validation of the added predictive value of a gene signature, i.e. to the verification that the inclusion of the new gene signature in a prediction model is able to improve its ...[more]

PMID: 25352096

Similar Datasets

Project description:Background: Lipidomics is emerging for biomarker discovery in cardiovascular disease, and circulating lipids are increasingly incorporated in risk models to predict cardiovascular events. Moreover, specific classes of lipids, such as sphingomyelins, ceramides, and triglycerides, have been related to coronary artery disease (CAD) severity and plaque characteristics. To avoid unnecessary testing, it is important to identify individuals at low CAD risk. The only pretest model available so far to rule out the presence of coronary atherosclerosis in patients with chest pain, but normal coronary arteries, is the minimal risk tool (MRT). Aim: Using state-of-the-art statistical methods, we aim to verify the additive predictive value of a set of lipids, derived from targeted plasma lipidomics of suspected CAD patients, to a re-estimated version of the MRT for ruling out the presence of coronary atherosclerosis assessed by coronary CT angiography (CCTA). Methods: Two hundred and fifty-six subjects with suspected stable CAD recruited from five European countries within H2020-SMARTool, undergoing CCTA and blood sampling for clinical biochemistry and lipidomics, were selected. The MRT was validated by regression methods and then re-estimated (reMRT). The reMRT was used as a baseline model in a likelihood ratio test approach to assess the added predictive value of each lipid from 13 among ceramides, triglycerides, and sphingomyelins. Except for one lipid, the analysis was carried out on more than 240 subjects for each lipid. A sensitivity analysis was carried out by considering two alternative models developed on the cohort as baseline models. Results: In 205 subjects, coronary atherosclerosis ranged from minimal lesions to overt obstructive CAD, while in 51 subjects (19.9%) the coronary arteries were intact. Four triglycerides and seven sphingomyelins were significantly (p < 0.05) and differentially expressed in the two groups and, at a lesser extent, one ceramide (p = 0.067). The probability of being at minimal risk was significantly better estimated by adding either Cer(d18:1/16:0) (p = 0.01), SM(40:2) (p = 0.04), or SM(41:1) at a lesser extent (p = 0.052) to reMRT than by applying the reMRT alone. The sensitivity analysis confirmed the relevance of these lipids. Furthermore, the addition of SM(34:1), SM(38:2), SM(41:2), and SM(42:4) improved the predictive performance of at least one of the other baseline models. None of the selected triglycerides was found to provide an added value. Conclusions: Plasma lipidomics can be a promising source of diagnostic and prognostic biomarkers in cardiovascular disease, exploitable not only to assess the risk of adverse events but also to identify subjects without coronary atherosclerosis, thus reducing unnecessary further testing in normal subjects.

Project description:BackgroundThe value of new biomarkers or imaging tests, when added to a prediction model, is currently evaluated using reclassification measures, such as the net reclassification improvement (NRI). However, these measures only provide an estimate of improved reclassification at population level. We present a straightforward approach to characterize subgroups of reclassified individuals in order to tailor implementation of a new prediction model to individuals expected to benefit from it.MethodsIn a large Dutch population cohort (n = 21,992) we classified individuals to low (< 5%) and high (? 5%) fatal cardiovascular disease risk by the Framingham risk score (FRS) and reclassified them based on the systematic coronary risk evaluation (SCORE). Subsequently, we characterized the reclassified individuals and, in case of heterogeneity, applied cluster analysis to identify and characterize subgroups. These characterizations were used to select individuals expected to benefit from implementation of SCORE.ResultsReclassification after applying SCORE in all individuals resulted in an NRI of 5.00% (95% CI [-0.53%; 11.50%]) within the events, 0.06% (95% CI [-0.08%; 0.22%]) within the nonevents, and a total NRI of 0.051 (95% CI [-0.004; 0.116]). Among the correctly downward reclassified individuals cluster analysis identified three subgroups. Using the characterizations of the typically correctly reclassified individuals, implementing SCORE only in individuals expected to benefit (n = 2,707,12.3%) improved the NRI to 5.32% (95% CI [-0.13%; 12.06%]) within the events, 0.24% (95% CI [0.10%; 0.36%]) within the nonevents, and a total NRI of 0.055 (95% CI [0.001; 0.123]). Overall, the risk levels for individuals reclassified by tailored implementation of SCORE were more accurate.DiscussionIn our empirical example the presented approach successfully characterized subgroups of reclassified individuals that could be used to improve reclassification and reduce implementation burden. In particular when newly added biomarkers or imaging tests are costly or burdensome such a tailored implementation strategy may save resources and improve (cost-)effectiveness.

Dataset Information

Added predictive value of omics data: specific issues related to validation illustrated by two case studies.

Publications

Added predictive value of omics data: specific issues related to validation illustrated by two case studies.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets