Dataset Information

Development and external validation study of a melanoma risk prediction model incorporating clinically assessed naevi and solar lentigines.

ABSTRACT:

Background

Melanoma risk prediction models could be useful for matching preventive interventions to patients' risk.

Objectives

To develop and validate a model for incident first-primary cutaneous melanoma using clinically assessed risk factors.

Methods

We used unconditional logistic regression with backward selection from the Australian Melanoma Family Study (461 cases and 329 controls) in which age, sex and city of recruitment were kept in each step, and we externally validated it using the Leeds Melanoma Case-Control Study (960 cases and 513 controls). Candidate predictors included clinically assessed whole-body naevi and solar lentigines, and self-assessed pigmentation phenotype, sun exposure, family history and history of keratinocyte cancer. We evaluated the predictive strength and discrimination of the model risk factors using odds per age- and sex-adjusted SD (OPERA) and the area under curve (AUC), and calibration using the Hosmer-Lemeshow test.

Results

The final model included the number of naevi ≥ 2 mm in diameter on the whole body, solar lentigines on the upper back (a six-level scale), hair colour at age 18 years and personal history of keratinocyte cancer. Naevi was the strongest risk factor; the OPERA was 3·51 [95% confidence interval (CI) 2·71-4·54] in the Australian study and 2·56 (95% CI 2·23-2·95) in the Leeds study. The AUC was 0·79 (95% CI 0·76-0·83) in the Australian study and 0·73 (95% CI 0·70-0·75) in the Leeds study. The Hosmer-Lemeshow test P-value was 0·30 in the Australian study and < 0·001 in the Leeds study.

Conclusions

This model had good discrimination and could be used by clinicians to stratify patients by melanoma risk for the targeting of preventive interventions. What's already known about this topic? Melanoma risk prediction models may be useful in prevention by tailoring interventions to personalized risk levels. For reasons of feasibility, time and cost many melanoma prediction models use self-assessed risk factors. However, individuals tend to underestimate their naevus numbers. What does this study add? We present a melanoma risk prediction model, which includes clinically-assessed whole-body naevi and solar lentigines, and self-assessed risk factors including pigmentation phenotype and history of keratinocyte cancer. This model performs well on discrimination, the model's ability to distinguish between individuals with and without melanoma, and may assist clinicians to stratify patients by melanoma risk for targeted preventive interventions.

SUBMITTER: Vuong K

PROVIDER: S-EPMC6997040 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:Background and objectivesThe International IgA Nephropathy Network recently developed and externally validated two models to predict the risk of progression of IgA nephropathy: full models without and with race. This study sought to externally validate the International IgA Nephropathy Prediction Tool in a large, independent, and contemporary cohort in China.Design, setting, participants, & measurementsWe included 1373 patients with biopsy-confirmed primary IgA nephropathy from The First Affiliated Hospital of Zhengzhou University from January 2012 to May 2018 and calculated predicted risks for each patient. The outcomes of interest were a 50% decline in eGFR or kidney failure. We assessed the performance of both models using discrimination (concordance statistics and Kaplan-Meier curves between subgroups), calibration (calibration plots), reclassification (net reclassification improvement and integrated discrimination improvement), and clinical utility (decision curve analysis).ResultsThe median follow-up was 29 months (interquartile range, 21-43 months; range, 1-95 months), and 186 (14%) patients reached the kidney outcomes of interest. Both models showed excellent discrimination (concordance statistics >0.85 and well separated survival curves). Overall, the full model without race generally underestimated the risk of primary outcome, whereas the full model with race was well calibrated for predicting 5-year risk. Compared with the full model without race, the full model with race had significant improvement in reclassification, as assessed by the net reclassification improvement (0.49; 95% confidence interval, 0.41 to 0.59) and integrated discrimination improvement (0.06; 95% confidence interval, 0.04 to 0.08). Decision curve analysis showed that both full models had a higher net benefit than default strategies, and the model with race performed better.ConclusionsIn this study, both full models demonstrated remarkable discrimination, acceptable calibration, and satisfactory clinical utility. The relatively short follow-up time may have limited the validation of these models.

Project description:IntroductionExternal validation studies are essential to study the generalizability of prediction models. Recently a permutation test, focusing on discrimination as quantified by the c-statistic, was proposed to judge whether a prediction model is transportable to a new setting. We aimed to evaluate this test and compare it to previously proposed procedures to judge any changes in c-statistic from development to external validation setting.MethodsWe compared the use of the permutation test to the use of benchmark values of the c-statistic following from a previously proposed framework to judge transportability of a prediction model. In a simulation study we developed a prediction model with logistic regression on a development set and validated them in the validation set. We concentrated on two scenarios: 1) the case-mix was more heterogeneous and predictor effects were weaker in the validation set compared to the development set, and 2) the case-mix was less heterogeneous in the validation set and predictor effects were identical in the validation and development set. Furthermore we illustrated the methods in a case study using 15 datasets of patients suffering from traumatic brain injury.ResultsThe permutation test indicated that the validation and development set were homogenous in scenario 1 (in almost all simulated samples) and heterogeneous in scenario 2 (in 17%-39% of simulated samples). Previously proposed benchmark values of the c-statistic and the standard deviation of the linear predictors correctly pointed at the more heterogeneous case-mix in scenario 1 and the less heterogeneous case-mix in scenario 2.ConclusionThe recently proposed permutation test may provide misleading results when externally validating prediction models in the presence of case-mix differences between the development and validation population. To correctly interpret the c-statistic found at external validation it is crucial to disentangle case-mix differences from incorrect regression coefficients.

Project description:BackgroundMany dementia prediction models have been developed, but only few have been externally validated, which hinders clinical uptake and may pose a risk if models are applied to actual patients regardless. Externally validating an existing prediction model is a difficult task, where we mostly rely on the completeness of model reporting in a published article. In this study, we aim to externally validate existing dementia prediction models. To that end, we define model reporting criteria, review published studies, and externally validate three well reported models using routinely collected health data from administrative claims and electronic health records.MethodsWe identified dementia prediction models that were developed between 2011 and 2020 and assessed if they could be externally validated given a set of model criteria. In addition, we externally validated three of these models (Walters' Dementia Risk Score, Mehta's RxDx-Dementia Risk Index, and Nori's ADRD dementia prediction model) on a network of six observational health databases from the United States, United Kingdom, Germany and the Netherlands, including the original development databases of the models.ResultsWe reviewed 59 dementia prediction models. All models reported the prediction method, development database, and target and outcome definitions. Less frequently reported by these 59 prediction models were predictor definitions (52 models) including the time window in which a predictor is assessed (21 models), predictor coefficients (20 models), and the time-at-risk (42 models). The validation of the model by Walters (development c-statistic: 0.84) showed moderate transportability (0.67-0.76 c-statistic). The Mehta model (development c-statistic: 0.81) transported well to some of the external databases (0.69-0.79 c-statistic). The Nori model (development AUROC: 0.69) transported well (0.62-0.68 AUROC) but performed modestly overall. Recalibration showed improvements for the Walters and Nori models, while recalibration could not be assessed for the Mehta model due to unreported baseline hazard.ConclusionWe observed that reporting is mostly insufficient to fully externally validate published dementia prediction models, and therefore, it is uncertain how well these models would work in other clinical settings. We emphasize the importance of following established guidelines for reporting clinical prediction models. We recommend that reporting should be more explicit and have external validation in mind if the model is meant to be applied in different settings.

Project description:Background and Objectives: The IDH (isocitrate dehydrogenase) status represents one of the main prognosis factors for gliomas. However, determining it requires invasive procedures and specialized surgical skills. Medical imaging such as MRI is essential in glioma diagnosis and management. Lately, fields such as Radiomics and Radiogenomics emerged as pertinent prediction tools for extracting molecular information out of medical images. These fields are based on Artificial Intelligence algorithms that require external validation in order to evaluate their general performance. The aim of this study was to provide an external validation for the algorithm formulated by Yoon Choi et al. of IDH status prediction using preoperative common MRI sequences and patient age. Material and Methods: We applied Choi's IDH status prediction algorithm on T1c, T2 and FLAIR preoperative MRI images of gliomas (grades WHO II-IV) of 21 operated adult patients from the Neurosurgery clinic of the Cluj County Emergency Clinical Hospital (CCECH), Cluj-Napoca Romania. We created a script to automate the testing process with DICOM format MRI sequences as input and IDH predicted status as output. Results: In terms of patient characteristics, the mean age was 48.6 ± 15.6; 57% were female and 43% male; 43% were IDH positive and 57% IDH negative. The proportions of WHO grades were 24%, 14% and 62% for II, III and IV, respectively. The validation test achieved a relative accuracy of 76% with 95% CI of (53%, 92%) and an Area Under the Curve (AUC) through DeLong et al. method of 0.74 with 95% CI of (0.53, 0.91) and a p of 0.021. Sensitivity and Specificity were 0.78 with 95% CI of (0.45, 0.96) and 0.75 with 95% CI of (0.47, 0.91), respectively. Conclusions: Although our results match the external test the author made on The Cancer Imaging Archive (TCIA) online dataset, performance of the algorithm on external data is still not high enough for clinical application. Radiogenomic approaches remain a high interest research field that may provide a rapid and accurate diagnosis and prognosis of patients with intracranial glioma.