Dataset Information

Evaluation of polygenic risk models using multiple performance measures: a critical assessment of discordant results.

ABSTRACT: PURPOSE:The area under the receiver operating characteristic curve (AUC) is commonly used for evaluating the improvement of polygenic risk models and increasingly assessed together with the net reclassification improvement (NRI) and integrated discrimination improvement (IDI). We evaluated how researchers described and interpreted AUC, NRI, and IDI when simultaneously assessed. METHODS:We reviewed how researchers described definitions of AUC, NRI, and IDI and how they computed each metric. Next, we reviewed how the increment in AUC, NRI, and IDI were interpreted, and how the overall conclusion about the improvement of the risk model was reached. RESULTS:AUC, NRI, and IDI were correctly defined in 63, 70, and 0% of the articles. All statistically significant values and almost half of the nonsignificant were interpreted as indicative of improvement, irrespective of the values of the metrics. Also, small, nonsignificant changes in the AUC were interpreted as indication of improvement when NRI and IDI were statistically significant. CONCLUSION:Researchers have insufficient knowledge about how to interpret the various metrics for the assessment of the predictive performance of polygenic risk models and rely on the statistical significance for their interpretation. A better understanding is needed to achieve more meaningful interpretation of polygenic prediction studies.

SUBMITTER: Martens FK

PROVIDER: S-EPMC6169739 | biostudies-literature | 2019 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Evaluation of polygenic risk models using multiple performance measures: a critical assessment of discordant results.

Martens Forike K FK Tonk Elisa C M ECM Janssens A Cecile J W ACJW

Genetics in medicine : official journal of the American College of Medical Genetics 20180612 2

<h4>Purpose</h4>The area under the receiver operating characteristic curve (AUC) is commonly used for evaluating the improvement of polygenic risk models and increasingly assessed together with the net reclassification improvement (NRI) and integrated discrimination improvement (IDI). We evaluated how researchers described and interpreted AUC, NRI, and IDI when simultaneously assessed.<h4>Methods</h4>We reviewed how researchers described definitions of AUC, NRI, and IDI and how they computed eac ...[more]

PMID: 29895851

Similar Datasets

Project description:BACKGROUND: Cancer survival studies are commonly analyzed using survival-time prediction models for cancer prognosis. A number of different performance metrics are used to ascertain the concordance between the predicted risk score of each patient and the actual survival time, but these metrics can sometimes conflict. Alternatively, patients are sometimes divided into two classes according to a survival-time threshold, and binary classifiers are applied to predict each patient's class. Although this approach has several drawbacks, it does provide natural performance metrics such as positive and negative predictive values to enable unambiguous assessments. METHODS: We compare the survival-time prediction and survival-time threshold approaches to analyzing cancer survival studies. We review and compare common performance metrics for the two approaches. We present new randomization tests and cross-validation methods to enable unambiguous statistical inferences for several performance metrics used with the survival-time prediction approach. We consider five survival prediction models consisting of one clinical model, two gene expression models, and two models from combinations of clinical and gene expression models. RESULTS: A public breast cancer dataset was used to compare several performance metrics using five prediction models. 1) For some prediction models, the hazard ratio from fitting a Cox proportional hazards model was significant, but the two-group comparison was insignificant, and vice versa. 2) The randomization test and cross-validation were generally consistent with the p-values obtained from the standard performance metrics. 3) Binary classifiers highly depended on how the risk groups were defined; a slight change of the survival threshold for assignment of classes led to very different prediction results. CONCLUSIONS: 1) Different performance metrics for evaluation of a survival prediction model may give different conclusions in its discriminatory ability. 2) Evaluation using a high-risk versus low-risk group comparison depends on the selected risk-score threshold; a plot of p-values from all possible thresholds can show the sensitivity of the threshold selection. 3) A randomization test of the significance of Somers' rank correlation can be used for further evaluation of performance of a prediction model. 4) The cross-validated power of survival prediction models decreases as the training and test sets become less balanced.

Project description:High levels of BCL-2 family proteins are implicated in a failed/ineffective apoptotic programme, often resulting in diseases, including cancer. Owing to their potential as drug targets in cancer therapy, several inhibitors of BCL-2 family proteins have been developed. These primarily target specific members of the BCL-2 family, particularly BCL-2 and BCL-XL but are ineffective against MCL-1. Major efforts have been invested in developing inhibitors of MCL-1, which is commonly amplified in human tumours and associated with tumour relapse and chemoresistance. In this report, the specificity of several BCL-2 family inhibitors (ABT-263, UCB-1350883, apogossypol and BH3I-1) was investigated and compared with putative MCL-1 inhibitors designed to exhibit improved or selective binding affinities for MCL-1 (TW-37, BI97C1, BI97C10, BI112D1, compounds 6 and 7, and MCL-1 inhibitor molecule (MIM-1)). ABT-263, BI97C1, BI112D1, MIM-1 and TW-37 exhibited specificity in inducing apoptosis in a Bax/Bak- and caspase-9-dependent manner, whereas the other agents showed no killing activity, or little or no specificity. Of these inhibitors, only ABT-263 and UCB-1350883 induced apoptosis in a BCL-2- or BCL-XL-dependent system. In cells that depend on MCL-1 for survival, ABT-263 and TW-37 induced extensive apoptosis, suggesting that at high concentrations these inhibitors have the propensity to inhibit MCL-1 in a cellular context. TW-37 induced apoptosis, assessed by chromatin condensation, caspase processing and phosphatidylserine externalisation, in a BAK-dependent manner and in cells that require MCL-1 for survival. TW-37-mediated apoptosis was also partly dependent on NOXA, suggesting that derivatives of TW-37, if engineered to exhibit better selectivity and efficacy at low nanomolar concentrations, may provide useful lead compounds for further synthetic programmes. Expanded medicinal chemistry iteration, as performed for the ABT series, may likewise improve the potency and specificity of the evaluated MCL-1 inhibitors.

Dataset Information

Evaluation of polygenic risk models using multiple performance measures: a critical assessment of discordant results.

Publications

Evaluation of polygenic risk models using multiple performance measures: a critical assessment of discordant results.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets