Visualizing Risk Prediction Models.
Ontology highlight
ABSTRACT: Objective
Risk prediction models can assist clinicians in making decisions. To boost the uptake of these models in clinical practice, it is important that end-users understand how the model works and can efficiently communicate its results. We introduce novel methods for interpretable model visualization.Methods
The proposed visualization techniques are applied to two prediction models from the Framingham Heart Study for the prediction of intermittent claudication and stroke after atrial fibrillation. We represent models using color bars, and visualize the risk estimation process for a specific patient using patient-specific contribution charts.Results
The color-based model representations provide users with an attractive tool to instantly gauge the relative importance of the predictors. The patient-specific representations allow users to understand the relative contribution of each predictor to the patient's estimated risk, potentially providing insightful information on which to base further patient management. Extensions towards non-linear models and interactions are illustrated on an artificial dataset.Conclusion
The proposed methods summarize risk prediction models and risk predictions for specific patients in an alternative way. These representations may facilitate communication between clinicians and patients.
SUBMITTER: Van Belle V
PROVIDER: S-EPMC4503430 | biostudies-literature | 2015
REPOSITORIES: biostudies-literature
Publications
<h4>Objective</h4>Risk prediction models can assist clinicians in making decisions. To boost the uptake of these models in clinical practice, it is important that end-users understand how the model works and can efficiently communicate its results. We introduce novel methods for interpretable model visualization.<h4>Methods</h4>The proposed visualization techniques are applied to two prediction models from the Framingham Heart Study for the prediction of intermittent claudication and stroke afte ...[more]
Similar Datasets
Project description:We propose and study two criteria to assess the usefulness of models that predict risk of disease incidence for screening and prevention, or the usefulness of prognostic models for management following disease diagnosis. The first criterion, the proportion of cases followed?PCF?(q), is the proportion of individuals who will develop disease who are included in the proportion?q?of individuals in the population at highest risk. The second criterion is the proportion needed to follow-up,?PNF?(p), namely the proportion of the general population at highest risk that one needs to follow in order that a proportion?p?of those destined to become cases will be followed.?PCF?(q)?assesses the effectiveness of a program that follows?100q% of the population at highest risk.?PNF?(p)?assess the feasibility of covering?100p% of cases by indicating how much of the population at highest risk must be followed. We show the relationship of those two criteria to the Lorenz curve and its inverse, and present distribution theory for estimates of?PCF?and?PNF. We develop new methods, based on influence functions, for inference for a single risk model, and also for comparing the?PCFs and?PNFs of two risk models, both of which were evaluated in the same validation data.
| S-EPMC3135785 | biostudies-literature
Project description:ImportanceThe clinical value of current multifactorial algorithms for individualized assessment of dementia risk remains unclear.ObjectiveTo evaluate the clinical value associated with 4 widely used dementia risk scores in estimating 10-year dementia risk.Design, setting, and participantsThis prospective population-based UK Biobank cohort study assessed 4 dementia risk scores at baseline (2006-2010) and ascertained incident dementia during the following 10 years. Replication with a 20-year follow-up was based on the British Whitehall II study. For both analyses, participants who had no dementia at baseline, had complete data on at least 1 dementia risk score, and were linked to electronic health records from hospitalizations or mortality were included. Data analysis was conducted from July 5, 2022, to April 20, 2023.ExposuresFour existing dementia risk scores: the Cardiovascular Risk Factors, Aging and Dementia (CAIDE)-Clinical score, the CAIDE-APOE-supplemented score, the Brief Dementia Screening Indicator (BDSI), and the Australian National University Alzheimer Disease Risk Index (ANU-ADRI).Main outcomes and measuresDementia was ascertained from linked electronic health records. To evaluate how well each score predicted the 10-year risk of dementia, concordance (C) statistics, detection rate, false-positive rate, and the ratio of true to false positives were calculated for each risk score and for a model including age alone.ResultsOf 465 929 UK Biobank participants without dementia at baseline (mean [SD] age, 56.5 [8.1] years; range, 38-73 years; 252 778 [54.3%] female participants), 3421 were diagnosed with dementia at follow-up (7.5 per 10 000 person-years). If the threshold for a positive test result was calibrated to achieve a 5% false-positive rate, all 4 risk scores detected 9% to 16% of incident dementia and therefore missed 84% to 91% (failure rate). The corresponding failure rate was 84% for a model that included age only. For a positive test result calibrated to detect at least half of future incident dementia, the ratio of true to false positives ranged between 1 to 66 (for CAIDE-APOE-supplemented) and 1 to 116 (for ANU-ADRI). For age alone, the ratio was 1 to 43. The C statistic was 0.66 (95% CI, 0.65-0.67) for the CAIDE clinical version, 0.73 (95% CI, 0.72-0.73) for the CAIDE-APOE-supplemented, 0.68 (95% CI, 0.67-0.69) for BDSI, 0.59 (95% CI, 0.58-0.60) for ANU-ADRI, and 0.79 (95% CI, 0.79-0.80) for age alone. Similar C statistics were seen for 20-year dementia risk in the Whitehall II study cohort, which included 4865 participants (mean [SD] age, 54.9 [5.9] years; 1342 [27.6%] female participants). In a subgroup analysis of same-aged participants aged 65 (±1) years, discriminatory capacity of risk scores was low (C statistics between 0.52 and 0.60).Conclusions and relevanceIn these cohort studies, individualized assessments of dementia risk using existing risk prediction scores had high error rates. These findings suggest that the scores were of limited value in targeting people for dementia prevention. Further research is needed to develop more accurate algorithms for estimation of dementia risk.
| S-EPMC10265307 | biostudies-literature
Project description:For predicting genetic risk, we propose a statistical approach that is specifically adapted to dealing with the challenges imposed by disease phenotypes and case-control sampling. Our approach (termed Genetic Risk Scores Inference [GeRSI]), combines the power of fixed-effects models (which estimate and aggregate the effects of single SNPs) and random-effects models (which rely primarily on whole-genome similarities between individuals) within the framework of the widely used liability-threshold model. We demonstrate in extensive simulation that GeRSI produces predictions that are consistently superior to current state-of-the-art approaches. When applying GeRSI to seven phenotypes from the Wellcome Trust Case Control Consortium (WTCCC) study, we confirm that the use of random effects is most beneficial for diseases that are known to be highly polygenic: hypertension (HT) and bipolar disorder (BD). For HT, there are no significant associations in the WTCCC data. The fixed-effects model yields an area under the ROC curve (AUC) of 54%, whereas GeRSI improves it to 59%. For BD, using GeRSI improves the AUC from 55% to 62%. For individuals ranked at the top 10% of BD risk predictions, using GeRSI substantially increases the BD relative risk from 1.4 to 2.5.
| S-EPMC4185122 | biostudies-literature
Project description:Case-cohort studies are increasingly used to quantify the association of novel factors with disease risk. Conventional measures of predictive ability need modification for this design. We show how Harrell's C-index, Royston's D, and the category-based and continuous versions of the net reclassification index (NRI) can be adapted.We simulated full cohort and case-cohort data, with sampling fractions ranging from 1% to 90%, using covariates from a cohort study of coronary heart disease, and two incidence rates. We then compared the accuracy and precision of the proposed risk prediction metrics.The C-index and D must be weighted in order to obtain unbiased results. The NRI does not need modification, provided that the relevant non-subcohort cases are excluded from the calculation. The empirical standard errors across simulations were consistent with analytical standard errors for the C-index and D but not for the NRI. Good relative efficiency of the prediction metrics was observed in our examples, provided the sampling fraction was above 40% for the C-index, 60% for D, or 30% for the NRI. Stata code is made available.Case-cohort designs can be used to provide unbiased estimates of the C-index, D measure and NRI.
| S-EPMC3848813 | biostudies-other
Project description:BACKGROUND:Clinical prediction models are useful in estimating a patient's risk of having a certain disease or experiencing an event in the future based on their current characteristics. Defining an appropriate risk threshold to recommend intervention is a key challenge in bringing a risk prediction model to clinical application; such risk thresholds are often defined in an ad hoc way. This is problematic because tacitly assumed costs of false positive and false negative classifications may not be clinically sensible. For example, when choosing the risk threshold that maximizes the proportion of patients correctly classified, false positives and false negatives are assumed equally costly. Furthermore, small to moderate sample sizes may lead to unstable optimal thresholds, which requires a particularly cautious interpretation of results. MAIN TEXT:We discuss how three common myths about risk thresholds often lead to inappropriate risk stratification of patients. First, we point out the contexts of counseling and shared decision-making in which a continuous risk estimate is more useful than risk stratification. Second, we argue that threshold selection should reflect the consequences of the decisions made following risk stratification. Third, we emphasize that there is usually no universally optimal threshold but rather that a plausible risk threshold depends on the clinical context. Consequently, we recommend to present results for multiple risk thresholds when developing or validating a prediction model. CONCLUSION:Bearing in mind these three considerations can avoid inappropriate allocation (and non-allocation) of interventions. Using discriminating and well-calibrated models will generate better clinical outcomes if context-dependent thresholds are used.
| S-EPMC6814132 | biostudies-literature
Project description:Alzheimer’s disease (AD) is the most common subtype of dementia, followed by Vascular Dementia (VaD), and Dementia with Lewy Bodies (DLB). Recently, microRNAs (miRNAs) have received a lot of attention as the novel biomarkers for dementia. Here, using serum miRNA expression of 1,601 Japanese individuals, we investigated potential miRNA bio- markers and constructed risk prediction models, based on a supervised principal component analysis (PCA) logistic regression method, according to the subtype of dementia. The final risk prediction model achieved a high accuracy of 0.873 on a validation cohort in AD, when using 78 miRNAs: Accuracy = 0.836 with 86 miRNAs in VaD; Accuracy = 0.825 with 110 miRNAs in DLB. To our knowledge, this is the first report applying miRNA-based risk pre- diction models to a dementia prospective cohort. Our study demonstrates our models to be effective in prospective disease risk prediction; and with further improvement may contribute to practical clinical use in dementia.
2019-02-26 | GSE120584 | GEO
Project description:There are numerous statistical models used to identify individuals at high risk of cancer due to inherited mutations. Mendelian models predict future risk of cancer by using family history with estimated cancer penetrances (age- and sex-specific risk of cancer given the genotype of the mutations) and mutation prevalences. However, there is often residual risk heterogeneity across families even after accounting for the mutations in the model, due to environmental or unobserved genetic risk factors. We aim to improve Mendelian risk prediction by incorporating a frailty model that contains a family-specific frailty vector, impacting the cancer hazard function, to account for this heterogeneity. We use a discrete uniform population frailty distribution and implement a marginalized approach that averages each family's risk predictions over the family's frailty distribution. We apply the proposed approach to improve breast cancer prediction in BRCAPRO, a Mendelian model that accounts for inherited mutations in the BRCA1 and BRCA2 genes to predict breast and ovarian cancer. We evaluate the proposed model's performance in simulations and real data from the Cancer Genetics Network and show improvements in model calibration and discrimination. We also discuss alternative approaches for incorporating frailties and their strengths and limitations.
| S-EPMC7895423 | biostudies-literature
Project description:Prediction models are commonly used to estimate risk for cardiovascular diseases, to inform diagnosis and management. However, performance may vary substantially across relevant subgroups of the population. Here we investigated heterogeneity of accuracy and fairness metrics across a variety of subgroups for risk prediction of two common diseases: atrial fibrillation (AF) and atherosclerotic cardiovascular disease (ASCVD). We calculated the Cohorts for Heart and Aging in Genomic Epidemiology Atrial Fibrillation (CHARGE-AF) score for AF and the Pooled Cohort Equations (PCE) score for ASCVD in three large datasets: Explorys Life Sciences Dataset (Explorys, n = 21,809,334), Mass General Brigham (MGB, n = 520,868), and the UK Biobank (UKBB, n = 502,521). Our results demonstrate important performance heterogeneity across subpopulations defined by age, sex, and presence of preexisting disease, with fairly consistent patterns across both scores. For example, using CHARGE-AF, discrimination declined with increasing age, with a concordance index of 0.72 [95% CI 0.72-0.73] for the youngest (45-54 years) subgroup to 0.57 [0.56-0.58] for the oldest (85-90 years) subgroup in Explorys. Even though sex is not included in CHARGE-AF, the statistical parity difference (i.e., likelihood of being classified as high risk) was considerable between males and females within the 65-74 years subgroup with a value of - 0.33 [95% CI - 0.33 to - 0.33]. We also observed weak discrimination (i.e., < 0.7) and suboptimal calibration (i.e., calibration slope outside of 0.7-1.3) in large subsets of the population; for example, all individuals aged 75 years or older in Explorys (17.4%). Our findings highlight the need to characterize and quantify the behavior of clinical risk models within specific subpopulations so they can be used appropriately to facilitate more accurate, consistent, and equitable assessment of disease risk.
| S-EPMC9307639 | biostudies-literature
Project description:ObjectivesTo look at the available literature on validated prediction models for contrast induced nephropathy and describe their characteristics.DesignSystematic review.Data sourcesMedline, Embase, and CINAHL (cumulative index to nursing and allied health literature) databases.Review methodsDatabases searched from inception to 2015, and the retrieved reference lists hand searched. Dual reviews were conducted to identify studies published in the English language of prediction models tested with patients that included derivation and validation cohorts. Data were extracted on baseline patient characteristics, procedural characteristics, modelling methods, metrics of model performance, risk of bias, and clinical usefulness. Eligible studies evaluated characteristics of predictive models that identified patients at risk of contrast induced nephropathy among adults undergoing a diagnostic or interventional procedure using conventional radiocontrast media (media used for computed tomography or angiography, and not gadolinium based contrast).Results16 studies were identified, describing 12 prediction models. Substantial interstudy heterogeneity was identified, as a result of different clinical settings, cointerventions, and the timing of creatinine measurement to define contrast induced nephropathy. Ten models were validated internally and six were validated externally. Discrimination varied in studies that were validated internally (C statistic 0.61-0.95) and externally (0.57-0.86). Only one study presented reclassification indices. The majority of higher performing models included measures of pre-existing chronic kidney disease, age, diabetes, heart failure or impaired ejection fraction, and hypotension or shock. No prediction model evaluated its effect on clinical decision making or patient outcomes.ConclusionsMost predictive models for contrast induced nephropathy in clinical use have modest ability, and are only relevant to patients receiving contrast for coronary angiography. Further research is needed to develop models that can better inform patient centred decision making, as well as improve the use of prevention strategies for contrast induced nephropathy.
| S-EPMC4784870 | biostudies-literature
Project description:ContextEarly detection of kidney cancer improves survival; however, low prevalence means that population-wide screening may be inefficient. Stratification of the population into risk categories could allow for the introduction of a screening programme tailored to individuals.ObjectiveThis review will identify and compare published models that predict the risk of developing kidney cancer in the general population.Evidence acquisitionA search identified primary research reporting or validating models predicting the risk of kidney cancer in Medline and EMBASE. After screening identified studies for inclusion, we extracted data onto a standardised form. The risk models were classified using the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines and evaluated using the PROBAST assessment tool.Evidence synthesisThe search identified 15 281 articles. Sixty-two satisfied the inclusion criteria; performance measures were provided for 11 models. Some models predicted the risk of prevalent undiagnosed disease and others future incident disease. Six of the models had been validated, two using external populations. The most commonly included risk factors were age, smoking status, and body mass index. Most of the models had acceptable-to-good discrimination (area under the receiver-operating curve >0.7) in development and validation. Many models also had high specificity; however, several had low sensitivity. The highest performance was seen for the models using only biomarkers to detect kidney cancer; however, these were developed and validated in small case-control studies.ConclusionsWe identified a small number of risk models that could be used to stratify the population according to the risk of kidney cancer. Most exhibit reasonable discrimination, but a few have been validated externally in population-based studies.Patient summaryIn this review, we looked at mathematical models predicting the likelihood of an individual developing kidney cancer. We found several suitable models, using a range of risk factors (such as age and smoking) to predict the risk for individuals. Most of the models identified require further testing in the general population to confirm their usefulness.
| S-EPMC8642244 | biostudies-literature