Dementia risk in the general population: large-scale external validation of prediction models in the AGES-Reykjavik study.
Ontology highlight
ABSTRACT:
SUBMITTER: Vonk JMJ
PROVIDER: S-EPMC8542560 | biostudies-literature | 2021 Oct
REPOSITORIES: biostudies-literature
Publications
Dementia risk in the general population: large-scale external validation of prediction models in the AGES-Reykjavik study.
Vonk Jet M J JMJ Greving Jacoba P JP Gudnason Vilmundur V Launer Lenore J LJ Geerlings Mirjam I MI
European journal of epidemiology 20210725 10
We aimed to evaluate the external performance of prediction models for all-cause dementia or AD in the general population, which can aid selection of high-risk individuals for clinical trials and prevention. We identified 17 out of 36 eligible published prognostic models for external validation in the population-based AGES-Reykjavik Study. Predictive performance was assessed with c statistics and calibration plots. All five models with a c statistic > .75 (.76-.81) contained cognitive testing as ...[more]
Similar Datasets
Project description:BackgroundMany dementia prediction models have been developed, but only few have been externally validated, which hinders clinical uptake and may pose a risk if models are applied to actual patients regardless. Externally validating an existing prediction model is a difficult task, where we mostly rely on the completeness of model reporting in a published article. In this study, we aim to externally validate existing dementia prediction models. To that end, we define model reporting criteria, review published studies, and externally validate three well reported models using routinely collected health data from administrative claims and electronic health records.MethodsWe identified dementia prediction models that were developed between 2011 and 2020 and assessed if they could be externally validated given a set of model criteria. In addition, we externally validated three of these models (Walters' Dementia Risk Score, Mehta's RxDx-Dementia Risk Index, and Nori's ADRD dementia prediction model) on a network of six observational health databases from the United States, United Kingdom, Germany and the Netherlands, including the original development databases of the models.ResultsWe reviewed 59 dementia prediction models. All models reported the prediction method, development database, and target and outcome definitions. Less frequently reported by these 59 prediction models were predictor definitions (52 models) including the time window in which a predictor is assessed (21 models), predictor coefficients (20 models), and the time-at-risk (42 models). The validation of the model by Walters (development c-statistic: 0.84) showed moderate transportability (0.67-0.76 c-statistic). The Mehta model (development c-statistic: 0.81) transported well to some of the external databases (0.69-0.79 c-statistic). The Nori model (development AUROC: 0.69) transported well (0.62-0.68 AUROC) but performed modestly overall. Recalibration showed improvements for the Walters and Nori models, while recalibration could not be assessed for the Mehta model due to unreported baseline hazard.ConclusionWe observed that reporting is mostly insufficient to fully externally validate published dementia prediction models, and therefore, it is uncertain how well these models would work in other clinical settings. We emphasize the importance of following established guidelines for reporting clinical prediction models. We recommend that reporting should be more explicit and have external validation in mind if the model is meant to be applied in different settings.
| S-EPMC9720950 | biostudies-literature
Project description:BackgroundEarly identification of dementia is crucial for prompt intervention for high-risk individuals in the general population. External validation studies on prognostic models for dementia have highlighted the need for updated models. The use of machine learning in dementia prediction is in its infancy and may improve predictive performance. The current study aimed to explore the difference in performance of machine learning algorithms compared to traditional statistical techniques, such as logistic and Cox regression, for prediction of all-cause dementia. Our secondary aim was to assess the feasibility of only using clinically accessible predictors rather than MRI predictors.MethodsData are from 4,793 participants in the population-based AGES-Reykjavik Study without dementia or mild cognitive impairment at baseline (mean age: 76 years, % female: 59%). Cognitive, biometric, and MRI assessments (total: 59 variables) were collected at baseline, with follow-up of incident dementia diagnoses for a maximum of 12 years. Machine learning algorithms included elastic net regression, random forest, support vector machine, and elastic net Cox regression. Traditional statistical methods for comparison were logistic and Cox regression. Model 1 was fit using all variables and model 2 was after feature selection using the Boruta package. A third model explored performance when leaving out neuroimaging markers (clinically accessible model). Ten-fold cross-validation, repeated ten times, was implemented during training. Upsampling was used to account for imbalanced data. Tuning parameters were optimized for recalibration automatically using the caret package in R.Results19% of participants developed all-cause dementia. Machine learning algorithms were comparable in performance to logistic regression in all three models. However, a slight added performance was observed in the elastic net Cox regression in the third model (c = 0.78, 95% CI: 0.78-0.78) compared to the traditional Cox regression (c = 0.75, 95% CI: 0.74-0.77).ConclusionsSupervised machine learning only showed added benefit when using survival techniques. Removing MRI markers did not significantly worsen our model's performance. Further, we presented the use of a nomogram using machine learning methods, showing transportability for the use of machine learning models in clinical practice. External validation is needed to assess the use of this model in other populations. Identifying high-risk individuals will amplify prevention efforts and selection for clinical trials.
| S-EPMC10463542 | biostudies-literature
Project description:ObjectiveTo determine whether microvascular damage, indicated by cerebral microbleeds (CMBs) and retinal microvascular signs, is associated with cognitive function and dementia in older persons.MethodsThis is a cross-sectional study of 3,906 participants (mean age 76 years; 58% women) in the AGES-Reykjavik Study (2002-2006). We assessed CMBs on MRI and retinal microvascular signs on digital retinal images. Composite Z scores of memory, processing speed, and executive function were derived from a battery of neurocognitive tests. Dementia and subtypes were diagnosed following international criteria. Regression models were used to relate cognitive Z scores and dementia to CMBs and retinal microvascular signs, adjusting for demographics, cardiovascular factors, and brain ischemic lesions.ResultsPeople with multiple (≥ 2) CMBs had lower Z scores on tests of processing speed (β-coefficient -0.16; 95% confidence interval -0.26 to -0.05) and executive function (-0.14; -0.24 to -0.04); results were strongest for having multiple CMBs located in the deep hemispheric or infratentorial areas. The odds ratio of vascular dementia was 2.32 (95% confidence interval 1.02 to 5.25) for multiple CMBs and 1.95 (1.04 to 3.62) for retinopathy. Having both CMBs and retinopathy, compared to having neither, was significantly associated with markedly slower processing speed (-0.25; -0.37 to -0.12), poorer executive function (-0.19; -0.31 to -0.07), and an increased odds ratio of vascular dementia (3.10; 1.11 to 8.62).ConclusionHaving multiple CMBs or concomitant CMBs and retinopathy is associated with a profile of vascular cognitive impairment. These findings suggest that microvascular damage, as indicated by CMBs and retinopathy lesions, has functional consequences in older men and women living in the community.
| S-EPMC3013588 | biostudies-literature
Project description:BACKGROUND AND PURPOSE:The differentiation of brain infarcts by region is important because their cause and clinical implications may differ. Information on the incidence of these lesions and association with cognition and dementia from longitudinal population studies is scarce. We investigated the incidence of infarcts in cortical, subcortical, cerebellar, and overall brain regions and how prevalent and incident infarcts associate with cognitive change and incident dementia. METHODS:Participants (n=2612, 41% men, mean age 74.6±4.8) underwent brain magnetic resonance imaging for the assessment of infarcts and cognitive testing at baseline and on average 5.2 years later. Incident dementia was assessed according to the international guidelines. RESULTS:Twenty-one percent of the study participants developed new infarcts. The risk of incident infarcts in men was higher than the risk in women (1.8; 95% confidence interval, 1.5-2.3). Persons with both incident and prevalent infarcts showed steeper cognitive decline and had almost double relative risk of incident dementia (1.7; 95% confidence interval, 1.3-2.2) compared with those without infarcts. Persons with new subcortical infarcts had the highest risk of incident dementia compared with those without infarcts (2.6; 95% confidence interval, 1.9-3.4). CONCLUSIONS:Men are at greater risk of developing incident brain infarcts than women. Persons with incident brain infarcts decline faster in cognition and have an increased risk of dementia compared with those free of infarcts. Incident subcortical infarcts contribute more than cortical and cerebellar infarcts to incident dementia which may indicate that infarcts of small vessel disease origin contribute more to the development of dementia than infarcts of embolic origin in larger vessels.
| S-EPMC5588878 | biostudies-literature
Project description:To systematically review the literature for dementia prediction models for use in the general population and externally validate their performance in a head-to-head comparison. We selected four prediction models for validation: CAIDE, BDSI, ANU-ADRI and DRS. From the Rotterdam Study, 6667 non-demented individuals aged 55 years and older were assessed between 1997 and 2001. Subsequently, participants were followed for dementia until 1 January, 2015. For each individual, we computed the risk of dementia using the reported scores from each prediction model. We used the C-statistic and calibration plots to assess the performance of each model to predict 10-year risk of all-cause dementia. For comparisons, we also evaluated discriminative accuracy using only the age component of these risk scores for each model separately. During 75,581 person-years of follow-up, 867 participants developed dementia. C-statistics for 10-year dementia risk prediction were 0.55 (95% CI 0.53-0.58) for CAIDE, 0.78 (0.76-0.81) for BDSI, 0.75 (0.74-0.77) for ANU-ADRI, and 0.81 (0.78-0.83) for DRS. Calibration plots showed that predicted risks were too extreme with underestimation at low risk and overestimation at high risk. Importantly, in all models age alone already showed nearly identical discriminative accuracy as the full model (C-statistics: 0.55 (0.53-0.58) for CAIDE, 0.81 (0.78-0.83) for BDSI, 0.77 (0.75-0.79) for ANU-ADRI, and 0.81 (0.78-0.83) for DRS). In this study, we found high variability in discriminative ability for predicting dementia in an elderly, community-dwelling population. All models showed similar discriminative ability when compared to prediction based on age alone. These findings highlight the urgent need for updated or new models to predict dementia risk in the general population.
| S-EPMC6061119 | biostudies-literature
Project description:BackgroundThere are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally tested, and how well they generally perform has not been broadly evaluated.MethodsA SCOPUS citation search was run on March 22, 2017 to identify external validations of cardiovascular CPMs in the Tufts Predictive Analytics and Comparative Effectiveness CPM Registry. We assessed the extent of external validation, performance heterogeneity across databases, and explored factors associated with model performance, including a global assessment of the clinical relatedness between the derivation and validation data.ResultsWe identified 2030 external validations of 1382 CPMs. Eight hundred seven (58%) of the CPMs in the Registry have never been externally validated. On average, there were 1.5 validations per CPM (range, 0-94). The median external validation area under the receiver operating characteristic curve was 0.73 (25th-75th percentile [interquartile range (IQR)], 0.66-0.79), representing a median percent decrease in discrimination of -11.1% (IQR, -32.4% to +2.7%) compared with performance on derivation data. 81% (n=1333) of validations reporting area under the receiver operating characteristic curve showed discrimination below that reported in the derivation dataset. 53% (n=983) of the validations report some measure of CPM calibration. For CPMs evaluated more than once, there was typically a large range of performance. Of 1702 validations classified by relatedness, the percent change in discrimination was -3.7% (IQR, -13.2 to 3.1) for closely related validations (n=123), -9.0 (IQR, -27.6 to 3.9) for related validations (n=862), and -17.2% (IQR, -42.3 to 0) for distantly related validations (n=717; P<0.001).ConclusionsMany published cardiovascular CPMs have never been externally validated, and for those that have, apparent performance during development is often overly optimistic. A single external validation appears insufficient to broadly understand the performance heterogeneity across different settings.
| S-EPMC8366535 | biostudies-literature
Project description:BackgroundExternal validations and comparisons of prognostic models or scores are a prerequisite for their use in routine clinical care but are lacking in most medical fields including chronic obstructive pulmonary disease (COPD). Our aim was to externally validate and concurrently compare prognostic scores for 3-year all-cause mortality in mostly multimorbid patients with COPD.MethodsWe relied on 24 cohort studies of the COPD Cohorts Collaborative International Assessment consortium, corresponding to primary, secondary, and tertiary care in Europe, the Americas, and Japan. These studies include globally 15,762 patients with COPD (1871 deaths and 42,203 person years of follow-up). We used network meta-analysis adapted to multiple score comparison (MSC), following a frequentist two-stage approach; thus, we were able to compare all scores in a single analytical framework accounting for correlations among scores within cohorts. We assessed transitivity, heterogeneity, and inconsistency and provided a performance ranking of the prognostic scores.ResultsDepending on data availability, between two and nine prognostic scores could be calculated for each cohort. The BODE score (body mass index, airflow obstruction, dyspnea, and exercise capacity) had a median area under the curve (AUC) of 0.679 [1st quartile-3rd quartile = 0.655-0.733] across cohorts. The ADO score (age, dyspnea, and airflow obstruction) showed the best performance for predicting mortality (difference AUCADO - AUCBODE = 0.015 [95% confidence interval (CI) = -0.002 to 0.032]; p = 0.08) followed by the updated BODE (AUCBODE updated - AUCBODE = 0.008 [95% CI = -0.005 to +0.022]; p = 0.23). The assumption of transitivity was not violated. Heterogeneity across direct comparisons was small, and we did not identify any local or global inconsistency.ConclusionsOur analyses showed best discriminatory performance for the ADO and updated BODE scores in patients with COPD. A limitation to be addressed in future studies is the extension of MSC network meta-analysis to measures of calibration. MSC network meta-analysis can be applied to prognostic scores in any medical field to identify the best scores, possibly paving the way for stratified medicine, public health, and research.
| S-EPMC5833113 | biostudies-literature
Project description:ObjectiveWe aimed to investigate the relation between apathy symptoms and structural brain changes on MRI, including white matter lesions (WMLs) and atrophy, in a large cohort of older persons.MethodsCross-sectional analyses are based on 4,354 persons without dementia (aged 76 ± 5 years) participating in the population-based Age, Gene/Environment Susceptibility-Reykjavik Study. Apathy symptoms were assessed with 3 items from the 15-item Geriatric Depression Scale. Brain volumes and total WML volume were estimated on 1.5-tesla MRI using an automated segmentation program; regional WML load was calculated using a semiquantitative scale. Regression analyses were adjusted for age, sex, education, intracranial volume, vascular risk factors, physical activity, brain infarcts, depressive symptoms, antidepressants, and cognitive status.ResultsCompared to those with <2 apathy symptoms, participants with ≥ 2 apathy symptoms (49% of the cohort) had significantly smaller gray matter volumes (mean adjusted difference -3.6 mL, 95% confidence interval [CI] -6.2 to -1.0), particularly in the frontal and temporal lobes; smaller white matter volumes (mean adjusted difference -1.9 mL, 95% CI -3.6 to -0.3), mainly in the parietal lobe; and smaller thalamus volumes. They were also more likely to have WMLs in the frontal lobe (adjusted odds ratio = 1.08, 95% CI 0.9-1.3). Excluding participants with a depression diagnosis did not change the associations.ConclusionsIn this older population without dementia, apathy symptoms are associated with a more diffuse loss of both gray and white matter volumes, independent of depression.
| S-EPMC4013817 | biostudies-literature
Project description:Imaging studies have reported conflicting findings on how brain structure differs with age and sex. This may be explained by discrepancies and limitations in study population and study design. We report a study on brain tissue volumes in one of the largest cohorts of individuals studied to date of subjects with high mean age (mean ± standard deviation (SD) 76 ± 6 years). These analyses are based on magnetic resonance imaging (MRI) scans acquired at baseline on 4303 non-demented elderly, and 367 who had a second MRI, on average 2.5 ± 0.2 years later. Tissue segmentation was performed with an automatic image analysis pipeline. Total brain parenchymal (TBP) volume decreased with increasing age while there was an increase in white matter hyperintensities (WMH) in both sexes. A reduction in both normal white matter (NWM)- and gray matter (GM) volume contributed to the brain shrinkage. After adjusting for intra-cranial volume, women had larger brain volumes compared to men (3.32%, p < 0.001) for TBP volume in the cross-sectional analysis. The longitudinal analysis showed a significant age-sex interaction in TBP volume with a greater rate of annual change in men (-0.70%, 95%CI: -0.78% to -0.63%) than women (-0.55%, 95%CI: -0.61% to -0.49%). The annual change in the cross-sectional data was approximately 40% less than the annual change in the longitudinal data and did not show significant age-sex interaction. The findings indicate that the cross-sectional data underestimate the rate of change in tissue volumes with age as the longitudinal data show greater rate of change in tissue volumes with age for all tissues.
| S-EPMC4712156 | biostudies-literature
Project description:IntroductionExternal validation studies are essential to study the generalizability of prediction models. Recently a permutation test, focusing on discrimination as quantified by the c-statistic, was proposed to judge whether a prediction model is transportable to a new setting. We aimed to evaluate this test and compare it to previously proposed procedures to judge any changes in c-statistic from development to external validation setting.MethodsWe compared the use of the permutation test to the use of benchmark values of the c-statistic following from a previously proposed framework to judge transportability of a prediction model. In a simulation study we developed a prediction model with logistic regression on a development set and validated them in the validation set. We concentrated on two scenarios: 1) the case-mix was more heterogeneous and predictor effects were weaker in the validation set compared to the development set, and 2) the case-mix was less heterogeneous in the validation set and predictor effects were identical in the validation and development set. Furthermore we illustrated the methods in a case study using 15 datasets of patients suffering from traumatic brain injury.ResultsThe permutation test indicated that the validation and development set were homogenous in scenario 1 (in almost all simulated samples) and heterogeneous in scenario 2 (in 17%-39% of simulated samples). Previously proposed benchmark values of the c-statistic and the standard deviation of the linear predictors correctly pointed at the more heterogeneous case-mix in scenario 1 and the less heterogeneous case-mix in scenario 2.ConclusionThe recently proposed permutation test may provide misleading results when externally validating prediction models in the presence of case-mix differences between the development and validation population. To correctly interpret the c-statistic found at external validation it is crucial to disentangle case-mix differences from incorrect regression coefficients.
| S-EPMC4755533 | biostudies-literature