Correction: External validation of risk prediction models for incident colorectal cancer using UK Biobank.
Ontology highlight
ABSTRACT:
SUBMITTER: Usher-Smith JA
PROVIDER: S-EPMC7217758 | biostudies-literature | 2020 May
REPOSITORIES: biostudies-literature
Publications
Correction: External validation of risk prediction models for incident colorectal cancer using UK Biobank.
Usher-Smith J A JA Harshfield A A Saunders C L CL Sharp S J SJ Emery J J Walter F M FM Muir K K Griffin S J SJ
British journal of cancer 20200501 10
An amendment to this paper has been published and can be accessed via a link at the top of the paper. ...[more]
Similar Datasets
Project description:BackgroundThis study aimed to compare and externally validate risk scores developed to predict incident colorectal cancer (CRC) that include variables routinely available or easily obtainable via self-completed questionnaire.MethodsExternal validation of fourteen risk models from a previous systematic review in 373 112 men and women within the UK Biobank cohort with 5-year follow-up, no prior history of CRC and data for incidence of CRC through linkage to national cancer registries.ResultsThere were 1719 (0.46%) cases of incident CRC. The performance of the risk models varied substantially. In men, the QCancer10 model and models by Tao, Driver and Ma all had an area under the receiver operating characteristic curve (AUC) between 0.67 and 0.70. Discrimination was lower in women: the QCancer10, Wells, Tao, Guesmi and Ma models were the best performing with AUCs between 0.63 and 0.66. Assessment of calibration was possible for six models in men and women. All would require country-specific recalibration if estimates of absolute risks were to be given to individuals.ConclusionsSeveral risk models based on easily obtainable data have relatively good discrimination in a UK population. Modelling studies are now required to estimate the potential health benefits and cost-effectiveness of implementing stratified risk-based CRC screening.
| S-EPMC5846069 | biostudies-literature
Project description:The aim of this study was to compare and externally validate risk scores developed to predict incident colorectal cancer that include common genetic variants (SNPs), with or without established lifestyle/environmental (questionnaire-based/classical/phenotypic) risk factors. We externally validated 23 risk models from a previous systematic review in 443,888 participants ages 37 to 73 from the UK Biobank cohort who had 6-year prospective follow-up, no prior history of colorectal cancer, and data for incidence of colorectal cancer through linkage to national cancer registries. There were 2,679 (0.6%) cases of incident colorectal cancer. We assessed model discrimination using the area under the operating characteristic curve (AUC) and relative risk calibration. The AUC of models including only SNPs increased with the number of included SNPs and was similar in men and women: the model by Huyghe with 120 SNPs had the highest AUC of 0.62 [95% confidence interval (CI), 0.59-0.64] in women and 0.64 (95% CI, 0.61-0.66) in men. Adding phenotypic risk factors without age improved discrimination in men but not in women. Adding phenotypic risk factors and age increased discrimination in all cases (P < 0.05), with the best performing models including SNPs, phenotypic risk factors, and age having AUCs between 0.64 and 0.67 in women and 0.67 and 0.71 in men. Relative risk calibration varied substantially across the models. Among middle-aged people in the UK, existing polygenic risk scores discriminate moderately well between those who do and do not develop colorectal cancer over 6 years. Consideration should be given to exploring the feasibility of incorporating genetic and lifestyle/environmental information in any future stratified colorectal cancer screening program.
| S-EPMC7610623 | biostudies-literature
Project description:BackgroundIn China, colorectal cancer (CRC) incidence and mortality have been steadily increasing over the last decades. Risk models to predict incident CRC have been developed in various populations, but they have not been systematically externally validated in a Chinese population. This study aimed to assess the performance of risk scores in predicting CRC using the China Kadoorie Biobank (CKB), one of the largest and geographically diverse prospective cohort studies in China.MethodsNine models were externally validated in 512,415 participants in CKB and included 2976 cases of CRC. Model discrimination was assessed, overall and by sex, age, site, and geographic location, using the area under the receiver operating characteristic curve (AUC). Model discrimination of these nine models was compared to a model using age alone. Calibration was assessed for five models, and they were re-calibrated in CKB.ResultsThe three models with the highest discrimination (Ma (Cox model) AUC 0.70 [95% CI 0.69-0.71]; Aleksandrova 0.70 [0.69-0.71]; Hong 0.69 [0.67-0.71]) included the variables age, smoking, and alcohol. These models performed significantly better than using a model based on age alone (AUC of 0.65 [95% CI 0.64-0.66]). Model discrimination was generally higher in younger participants, males, urban environments, and for colon cancer. The two models (Guo and Chen) developed in Chinese populations did not perform better than the others. Among the 10% of participants with the highest risk, the three best performing models identified 24-26% of participants that went on to develop CRC.ConclusionsSeveral risk models based on easily obtainable demographic and modifiable lifestyle factor have good discrimination in a Chinese population. The three best performing models have a higher discrimination than using a model based on age alone.
| S-EPMC9454206 | biostudies-literature
Project description:IntroductionRisk prediction is important for preventing and managing cardiovascular disease (CVD). CVD risk prediction tools designed for the general population may be inaccurate in people with inflammatory diseases.ObjectivesTo investigate the performance of four cardiovascular risk prediction tools (QRISK3, Framingham Risk Score, Reynolds Risk Score and SCORE) in psoriatic arthritis (PsA) and psoriasis. We also compare performance in participants with no inflammatory conditions and in people with rheumatoid arthritis (RA).MethodsThis research utilised the UK Biobank Resource. We identified participants with PsA, psoriasis and RA and calculated their cardiovascular risk using each risk tool. We assessed model calibration by comparing observed and predicted outcomes. Discrimination of 10-year risk prediction was assessed using time-dependent area under ROC curve (AUC), sensitivity, specificity, positive and negative predictive values.ResultsWe included 769 individuals with PsA, 8062 with psoriasis and 4772 with RA when assessing the QRISK3 tool. Predictions for individuals with psoriasis were roughly as accurate as those with no inflammatory conditions with time-dependent AUC of 0.74 (95%CI, 0.72, 0.76) and of 0.74 (95%CI, 0.72, 0.77) respectively. In contrast, individuals with PsA obtained the least accurate predictions with an AUC of 0.70 (95%CI, 0.64, 0.76). Individuals with RA also obtained less accurate predictions with AUC of 0.72 (0.69,0.74). For the Framingham risk score, AUCs varied between 0.61 (95%CI, 0.55, 0.68) for participants with PsA and 0.71 (95%CI, 0.68, 0.74) for individuals with no inflammatory condition.ConclusionsIn general, CVD risk prediction accuracy was similar for individuals with psoriasis or no inflammatory condition, but lower for individuals with PsA or RA.
| S-EPMC11865138 | biostudies-literature
Project description:BackgroundA previously developed risk prediction model needs to be validated before being used in a new population. The finite size of the validation sample entails that there is uncertainty around model performance. We apply value-of-information (VoI) methodology to quantify the consequence of uncertainty in terms of net benefit (NB).MethodsWe define the expected value of perfect information (EVPI) for model validation as the expected loss in NB due to not confidently knowing which of the alternative decisions confers the highest NB. We propose bootstrap-based and asymptotic methods for EVPI computations and conduct simulation studies to compare their performance. In a case study, we use the non-US subsets of a clinical trial as the development sample for predicting mortality after myocardial infarction and calculate the validation EVPI for the US subsample.ResultsThe computation methods generated similar EVPI values in simulation studies. EVPI generally declined with larger samples. In the case study, at the prespecified threshold of 0.02, the best decision with current information would be to use the model, with an incremental NB of 0.0020 over treating all. At this threshold, the EVPI was 0.0005 (relative EVPI = 25%). When scaled to the annual number of heart attacks in the US, the expected NB loss due to uncertainty was equal to 400 true positives or 19,600 false positives, indicating the value of further model validation.ConclusionVoI methods can be applied to the NB calculated during external validation of clinical prediction models. While uncertainty does not directly affect the clinical implications of NB findings, validation EVPI provides an objective perspective to the need for further validation and can be reported alongside NB in external validation studies.HighlightsExternal validation is a critical step when transporting a risk prediction model to a new setting, but the finite size of the validation sample creates uncertainty about the performance of the model.In decision theory, such uncertainty is associated with loss of net benefit because it can prevent one from identifying whether the use of the model is beneficial over alternative strategies.We define the expected value of perfect information for external validation as the expected loss in net benefit by not confidently knowing if the use of the model is net beneficial.The adoption of a model for a new population should be based on its expected net benefit; independently, value-of-information methods can be used to decide whether further validation studies are warranted.
| S-EPMC10336716 | biostudies-literature
Project description:Concordance measures are frequently used for assessing the discriminative ability of risk prediction models. The interpretation of estimated concordance at external validation is difficult if the case-mix differs from the model development setting. We aimed to develop a concordance measure that provides insight into the influence of case-mix heterogeneity and is robust to censoring of time-to-event data. We first derived a model-based concordance (mbc) measure that allows for quantification of the influence of case-mix heterogeneity on discriminative ability of proportional hazards and logistic regression models. This mbc can also be calculated including a regression slope that calibrates the predictions at external validation (c-mbc), hence assessing the influence of overall regression coefficient validity on discriminative ability. We derived variance formulas for both mbc and c-mbc. We compared the mbc and the c-mbc with commonly used concordance measures in a simulation study and in two external validation settings. The mbc was asymptotically equivalent to a previously proposed resampling-based case-mix corrected c-index. The c-mbc remained stable at the true value with increasing proportions of censoring, while Harrell's c-index and to a lesser extent Uno's concordance measure increased unfavorably. Variance estimates of mbc and c-mbc were well in agreement with the simulated empirical variances. We conclude that the mbc is an attractive closed-form measure that allows for a straightforward quantification of the expected change in a model's discriminative ability due to case-mix heterogeneity. The c-mbc also reflects regression coefficient validity and is a censoring-robust alternative for the c-index when the proportional hazards assumption holds. Copyright © 2016 John Wiley & Sons, Ltd.
| S-EPMC5550798 | biostudies-literature
Project description:BackgroundPeople with intermediate hyperglycemia (IH), including impaired fasting glucose and/or impaired glucose tolerance, are at higher risk of developing type 2 diabetes (T2D) than those with normoglycemia. We aimed to evaluate the performance of published T2D risk prediction models in Chinese people with IH to inform them about the choice of primary diabetes prevention measures.MethodsA systematic literature search was conducted to identify Asian-derived T2D risk prediction models, which were eligible if they were built on a prospective cohort of Asian adults without diabetes at baseline and utilized routinely-available variables to predict future risk of T2D. These Asian-derived and five prespecified non-Asian derived T2D risk prediction models were divided into BASIC (clinical variables only) and EXTENDED (plus laboratory variables) versions, with validation performed on them in three prospective Chinese IH cohorts: ACE (n = 3241), Luzhou (n = 1333), and TCLSIH (n = 1702). Model performance was assessed in terms of discrimination (C-statistic) and calibration (Hosmer-Lemeshow test).ResultsForty-four Asian and five non-Asian studies comprising 21 BASIC and 46 EXTENDED T2D risk prediction models for validation were identified. The majority were at high (n = 43, 87.8%) or unclear (n = 3, 6.1%) risk of bias, while only three studies (6.1%) were scored at low risk of bias. BASIC models showed poor-to-moderate discrimination with C-statistics 0.52-0.60, 0.50-0.59, and 0.50-0.64 in the ACE, Luzhou, and TCLSIH cohorts respectively. EXTENDED models showed poor-to-acceptable discrimination with C-statistics 0.54-0.73, 0.52-0.67, and 0.59-0.78 respectively. Fifteen BASIC and 40 EXTENDED models showed poor calibration (P < 0.05), overpredicting or underestimating the observed diabetes risk. Most recalibrated models showed improved calibration but modestly-to-severely overestimated diabetes risk in the three cohorts. The NAVIGATOR model showed the best discrimination in the three cohorts but had poor calibration (P < 0.05).ConclusionsIn Chinese people with IH, previously published BASIC models to predict T2D did not exhibit good discrimination or calibration. Several EXTENDED models performed better, but a robust Chinese T2D risk prediction tool in people with IH remains a major unmet need.
| S-EPMC9472437 | biostudies-literature
Project description:BackgroundThe clinical validity of the multifactorial BOADICEA model for epithelial tubo-ovarian cancer (EOC) risk prediction has not been assessed in a large sample size or over a longer term.MethodsWe evaluated the model discrimination and calibration in the UK Biobank cohort comprising 199,429 women (733 incident EOCs) of European ancestry without previous cancer history. We predicted 10-year EOC risk incorporating data on questionnaire-based risk factors (QRFs), family history, a 36-SNP polygenic risk score and pathogenic variants (PV) in six EOC susceptibility genes (BRCA1, BRCA2, RAD51C, RAD51D, BRIP1 and PALB2).ResultsDiscriminative ability was maximised under the multifactorial model that included all risk factors (AUC = 0.68, 95% CI: 0.66-0.70). This model was well calibrated in deciles of predicted risk with calibration slope=0.99 (95% CI: 0.98-1.01). Discriminative ability was similar in women younger or older than 60 years. The AUC was higher when analyses were restricted to PV carriers (0.76, 95% CI: 0.69-0.82). Using relative risk (RR) thresholds, the full model classified 97.7%, 1.7%, 0.4% and 0.2% women in the RR < 2.0, 2.0 ≤ RR < 2.9, 2.9 ≤ RR < 6.0 and RR ≥ 6.0 categories, respectively, identifying 9.1 of incident EOC among those with RR ≥ 2.0.DiscussionBOADICEA, implemented in CanRisk ( www.canrisk.org ), provides valid 10-year EOC risks and can facilitate clinical decision-making in EOC risk management.
| S-EPMC11519606 | biostudies-literature
Project description:Colorectal cancer (CRC) risk prediction models could be used to risk-stratify the population to provide individually tailored screening provision. Using participants from the UK Biobank prospective cohort study, we evaluated whether the addition of a genetic risk score (GRS) could improve the performance of two previously validated models. Inclusion of the GRS did not appreciably improve discrimination of either model, and led to substantial miscalibration. Following recalibration the discrimination did not change, but good calibration for models incorporating the GRS was recovered. Comparing predictions between models with and without the GRS, 5% of participants or fewer changed their absolute risk by ±0.3% or more in either model. In summary, addition of a GRS did not meaningfully improve the performance of validated CRC-risk prediction models. At present, provision of genetic information is not useful for risk stratification for CRC.
| S-EPMC6203780 | biostudies-literature
Project description:The value of a developed prediction model depends on its performance outside the development sample. The key is therefore to externally validate the model on a different but related independent data. In this study, we propose a permutation method to assess heterogeneity in external validation for risk prediction models. The permutation p value measures the extent of homology between development and validation datasets. If p < 0.05, the model may not be directly transported to the external validation population without further revision or updating. Monte-Carlo simulations are conducted to evaluate the statistical properties of the proposed method, and two microarray breast cancer datasets are analyzed for demonstration. The permutation method is easy to implement and is recommended for routine use in external validation for risk prediction models.
| S-EPMC4301917 | biostudies-literature