Dataset Information

Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators.

ABSTRACT: CONTEXT:Urologists regularly develop clinical risk prediction models to support clinical decisions. In contrast to traditional performance measures, decision curve analysis (DCA) can assess the utility of models for decision making. DCA plots net benefit (NB) at a range of clinically reasonable risk thresholds. OBJECTIVE:To provide recommendations on interpreting and reporting DCA when evaluating prediction models. EVIDENCE ACQUISITION:We informally reviewed the urological literature to determine investigators' understanding of DCA. To illustrate, we use data from 3616 patients to develop risk models for high-grade prostate cancer (n=313, 9%) to decide who should undergo a biopsy. The baseline model includes prostate-specific antigen and digital rectal examination; the extended model adds two predictors based on transrectal ultrasound (TRUS). EVIDENCE SYNTHESIS:We explain risk thresholds, NB, default strategies (treat all, treat no one), and test tradeoff. To use DCA, first determine whether a model is superior to all other strategies across the range of reasonable risk thresholds. If so, that model appears to improve decisions irrespective of threshold. Second, consider if there are important extra costs to using the model. If so, obtain the test tradeoff to check whether the increase in NB versus the best other strategy is worth the additional cost. In our case study, addition of TRUS improved NB by 0.0114, equivalent to 1.1 more detected high-grade prostate cancers per 100 patients. Hence, adding TRUS would be worthwhile if we accept subjecting 88 patients to TRUS to find one additional high-grade prostate cancer or, alternatively, subjecting 10 patients to TRUS to avoid one unnecessary biopsy. CONCLUSIONS:The proposed guidelines can help researchers understand DCA and improve application and reporting. PATIENT SUMMARY:Decision curve analysis can identify risk models that can help us make better clinical decisions. We illustrate appropriate reporting and interpretation of decision curve analysis.

SUBMITTER: Van Calster B

PROVIDER: S-EPMC6261531 | biostudies-literature | 2018 Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators.

Van Calster Ben B Wynants Laure L Verbeek Jan F M JFM Verbakel Jan Y JY Christodoulou Evangelia E Vickers Andrew J AJ Roobol Monique J MJ Steyerberg Ewout W EW

European urology 20180919 6

<h4>Context</h4>Urologists regularly develop clinical risk prediction models to support clinical decisions. In contrast to traditional performance measures, decision curve analysis (DCA) can assess the utility of models for decision making. DCA plots net benefit (NB) at a range of clinically reasonable risk thresholds.<h4>Objective</h4>To provide recommendations on interpreting and reporting DCA when evaluating prediction models.<h4>Evidence acquisition</h4>We informally reviewed the urological ...[more]

PMID: 30241973

Similar Datasets

Project description:BackgroundRisk prediction models have been proposed for various diseases and are being improved as new predictors are identified. A major challenge is to determine whether the newly discovered predictors improve risk prediction. Decision curve analysis has been proposed as an alternative to the area under the curve and net reclassification index to evaluate the performance of prediction models in clinical scenarios. The decision curve computed using the net benefit can evaluate the predictive performance of risk models at a given or range of threshold probabilities. However, when the decision curves for 2 competing models cross in the range of interest, it is difficult to identify the best model as there is no readily available summary measure for evaluating the predictive performance. The key deterrent for using simple measures such as the area under the net benefit curve is the assumption that the threshold probabilities are uniformly distributed among patients.MethodsWe propose a novel measure for performing decision curve analysis. The approach estimates the distribution of threshold probabilities without the need of additional data. Using the estimated distribution of threshold probabilities, the weighted area under the net benefit curve serves as the summary measure to compare risk prediction models in a range of interest.ResultsWe compared 3 different approaches, the standard method, the area under the net benefit curve, and the weighted area under the net benefit curve. Type 1 error and power comparisons demonstrate that the weighted area under the net benefit curve has higher power compared to the other methods. Several simulation studies are presented to demonstrate the improvement in model comparison using the weighted area under the net benefit curve compared to the standard method.ConclusionsThe proposed measure improves decision curve analysis by using the weighted area under the curve and thereby improves the power of the decision curve analysis to compare risk prediction models in a clinical scenario.

Project description:BackgroundDecision curve analysis can be used to determine whether a personalized model for treatment benefit would lead to better clinical decisions. Decision curve analysis methods have been described to estimate treatment benefit using data from a single randomized controlled trial.ObjectivesOur main objective is to extend the decision curve analysis methodology to the scenario in which several treatment options exist and evidence about their effects comes from a set of trials, synthesized using network meta-analysis (NMA).MethodsWe describe the steps needed to estimate the net benefit of a prediction model using evidence from studies synthesized in an NMA. We show how to compare personalized versus one-size-fit-all treatment decision-making strategies, such as "treat none" or "treat all patients with a specific treatment" strategies. First, threshold values for each included treatment need to be defined (i.e., the minimum risk difference compared with control that renders a treatment worth taking). The net benefit per strategy can then be plotted for a plausible range of threshold values to reveal the most clinically useful strategy. We applied our methodology to an NMA prediction model for relapsing-remitting multiple sclerosis, which can be used to choose between natalizumab, dimethyl fumarate, glatiramer acetate, and placebo.ResultsWe illustrated the extended decision curve analysis methodology using several threshold value combinations for each available treatment. For the examined threshold values, the "treat patients according to the prediction model" strategy performs either better than or close to the one-size-fit-all treatment strategies. However, even small differences may be important in clinical decision making. As the advantage of the personalized model was not consistent across all thresholds, improving the existing model (by including, for example, predictors that will increase discrimination) is needed before advocating its clinical usefulness.ConclusionsThis novel extension of decision curve analysis can be applied to NMA-based prediction models to evaluate their use to aid treatment decision making.HighlightsDecision curve analysis is extended into a (network) meta-analysis framework.Personalized models predicting treatment benefit are evaluated when several treatment options are available and evidence about their effects comes from a set of trials.Detailed steps to compare personalized versus one-size-fit-all treatment decision-making strategies are outlined.This extension of decision curve analysis can be applied to (network) meta-analysis-based prediction models to evaluate their use to aid treatment decision making.

Project description:BackgroundNon-alcoholic fatty liver disease (NAFLD) is the commonest liver condition in the western world and is directly linked to obesity and the metabolic syndrome. Elevated body mass index is regarded as a major risk factor of NAFL (steatosis) and NAFLD fibrosis. Using data from the Avon Longitudinal Study of Parents and Children (ALSPAC), we sought to investigate whether other variables from adolescence could improve prediction of future NAFL and NAFLD fibrosis risk at 24 years, above BMI and sex.MethodsAged 24 years, 4018 ALSPAC participants had transient elastography (TE) and controlled attenuation parameter (CAP) measurement using Echosens 502 Touch. 513 participants with harmful alcohol consumption were excluded. Logistic regression models examined which variables measured at 17 years were predictive of NAFL and NAFLD fibrosis in young adults. Predictors included sex, BMI, central adiposity, lipid profile, blood pressure, liver function tests, homeostatic model assessment for insulin resistance (HOMA-IR), and ultrasound defined NAFL at 17 years (when examining fibrosis outcomes). A model including all these variables was termed "routine clinical measures". Models were compared using area under the receiver operator curve (AUROC) and Bayesian Information Criterion (BIC), analysis, which penalises model complexity. Models were tested in all participants and those with overweight or obese standardised BMIs (BMI SDS) centiles at the 17-year time point. A decision curve analysis (DCA) was performed to evaluate the clinical utility of models in overweight and obese adolescents predicting NAFLD fibrosis at a threshold probability of 0.1.ResultsThe "routine clinical measures" model had the highest AUROC for predicting NAFL in all adolescent participants (AUROC 0.79 [SD 0.00]) and those with an overweight/obese BMI SDS centile (AUROC 0.77 [SD 0.01]). According to BIC analysis, insulin resistance was the best predictor of NAFL in all adolescents, whilst central adiposity was the best predictor in those with an overweight/obese BMI SDS centile. The "routine clinical measures" model also had the highest AUROC for predicting NAFLD fibrosis in all adolescent participants (AUROC 0.78 [SD 0.02]) and participants with an overweight/obese BMI SDS centile (AUROC 0.84 [SD 0.03]). However, following BIC analysis, BMI was the best predictor of NAFLD fibrosis in all adolescents including those with an overweight/obese BMI SDS centile. A decision curve analysis examining overweight/obese adolescent participants showed the model that had the greatest net benefit for increased NAFLD fibrosis detection, above a treat all overweight and obese adolescents' assumption, was the "routine clinical measures" model. However, the net benefit was marginal (0.0054 [0.0034-0.0075]).ConclusionIn adolescents, routine clinical measures were not superior to central adiposity and BMI at predicting NAFL and NAFLD fibrosis respectively in young adulthood. Additional routine clinical measurements do provide incremental benefit in detecting true positive fibrosis cases, but the benefit is small. Thus, to reduce morbidity and mortality associated with NASH cirrhosis in adults, the ultimate end point of NAFLD, the focus must be on obesity management at a population level.

Project description:BackgroundNext-generation sequencing (NGS) is an efficient tool used for identifying pathogenic variants that cause Mendelian disorders. However, the lack of bioinformatics training of researchers makes the interpretation of identified variants a challenge in terms of precision and efficiency. In addition, the non-standardized phenotypic description of human diseases also makes it difficult to establish an integrated analysis pathway for variant annotation and interpretation. Solutions to these bottlenecks are urgently needed.ResultsWe develop a tool named "Cruxome" to automatically annotate and interpret single nucleotide variants (SNVs) and small insertions and deletions (InDels). Our approach greatly simplifies the current burdensome task of clinical geneticists and scientists to identify the causative pathogenic variants and build personal knowledge reference bases. The integrated architecture of Cruxome offers key advantages such as an interactive and user-friendly interface and the assimilation of electronic health records of the patient. By combining a natural language processing algorithm, Cruxome can efficiently process the clinical description of diseases to HPO standardized vocabularies. By using machine learning, in silico predictive algorithms, integrated multiple databases and supplementary tools, Cruxome can automatically process SNVs and InDels variants (trio-family or proband-only cases) and clinical diagnosis records, then annotate, score, identify and interpret pathogenic variants to finally generate a standardized clinical report following American College of Medical Genetics and Genomics/ Association for Molecular Pathology (ACMG/AMP) guidelines. Cruxome also provides supplementary tools to examine and visualize the genes or variations in historical cases, which can help to better understand the genetic basis of the disease.ConclusionsCruxome is an efficient tool for annotation and interpretation of variations and dramatically reduces the workload for clinical geneticists and researchers to interpret NGS results, simplifying their decision-making processes. We present an online version of Cruxome, which is freely available to academics and clinical researchers. The site is accessible at http://114.251.61.49:10024/cruxome/ .

Dataset Information

Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators.

Publications

Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets