Dataset Information

Performance of a deep neural network in teledermatology: a single-centre prospective diagnostic study.

ABSTRACT:

Background

The use of artificial intelligence (AI) algorithms for the diagnosis of skin diseases has shown promise in experimental settings but has not been yet tested in real-life conditions.

Objective

To assess the diagnostic performance and potential clinical utility of a 174-multiclass AI algorithm in a real-life telemedicine setting.

Methods

Prospective, diagnostic accuracy study including consecutive patients who submitted images for teledermatology evaluation. The treating dermatologist chose a single image to upload to a web application during teleconsultation. A follow-up reader study including nine healthcare providers (3 dermatologists, 3 dermatology residents and 3 general practitioners) was performed.

Results

A total of 340 cases from 281 patients met study inclusion criteria. The mean (SD) age of patients was 33.7 (17.5) years; 63% (n = 177) were female. Exposure to the AI algorithm results was considered useful in 11.8% of visits (n = 40) and the teledermatologist correctly modified the real-time diagnosis in 0.6% (n = 2) of cases. The overall top-1 accuracy of the algorithm (41.2%) was lower than that of the dermatologists (60.1%), residents (57.8%) and general practitioners (49.3%) (all comparisons P < 0.05, in the reader study). When the analysis was limited to the diagnoses on which the algorithm had been explicitly trained, the balanced top-1 accuracy of the algorithm (47.6%) was comparable to the dermatologists (49.7%) and residents (47.7%) but superior to the general practitioners (39.7%; P = 0.049). Algorithm performance was associated with patient skin type and image quality.

Conclusions

A 174-disease class AI algorithm appears to be a promising tool in the triage and evaluation of lesions with patient-taken photographs via telemedicine.

SUBMITTER: Munoz-Lopez C

PROVIDER: S-EPMC8274350 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:ImportanceDeep learning convolutional neural networks (CNNs) have shown a performance at the level of dermatologists in the diagnosis of melanoma. Accordingly, further exploring the potential limitations of CNN technology before broadly applying it is of special interest.ObjectiveTo investigate the association between gentian violet surgical skin markings in dermoscopic images and the diagnostic performance of a CNN approved for use as a medical device in the European market.Design and settingA cross-sectional analysis was conducted from August 1, 2018, to November 30, 2018, using a CNN architecture trained with more than 120 000 dermoscopic images of skin neoplasms and corresponding diagnoses. The association of gentian violet skin markings in dermoscopic images with the performance of the CNN was investigated in 3 image sets of 130 melanocytic lesions each (107 benign nevi, 23 melanomas).ExposuresThe same lesions were sequentially imaged with and without the application of a gentian violet surgical skin marker and then evaluated by the CNN for their probability of being a melanoma. In addition, the markings were removed by manually cropping the dermoscopic images to focus on the melanocytic lesion.Main outcomes and measuresSensitivity, specificity, and area under the curve (AUC) of the receiver operating characteristic (ROC) curve for the CNN's diagnostic classification in unmarked, marked, and cropped images.ResultsIn all, 130 melanocytic lesions (107 benign nevi and 23 melanomas) were imaged. In unmarked lesions, the CNN achieved a sensitivity of 95.7% (95% CI, 79%-99.2%) and a specificity of 84.1% (95% CI, 76.0%-89.8%). The ROC AUC was 0.969. In marked lesions, an increase in melanoma probability scores was observed that resulted in a sensitivity of 100% (95% CI, 85.7%-100%) and a significantly reduced specificity of 45.8% (95% CI, 36.7%-55.2%, P < .001). The ROC AUC was 0.922. Cropping images led to the highest sensitivity of 100% (95% CI, 85.7%-100%), specificity of 97.2% (95% CI, 92.1%-99.0%), and ROC AUC of 0.993. Heat maps created by vanilla gradient descent backpropagation indicated that the blue markings were associated with the increased false-positive rate.Conclusions and relevanceThis study's findings suggest that skin markings significantly interfered with the CNN's correct diagnosis of nevi by increasing the melanoma probability scores and consequently the false-positive rate. A predominance of skin markings in melanoma training images may have induced the CNN's association of markings with a melanoma diagnosis. Accordingly, these findings suggest that skin markings should be avoided in dermoscopic images intended for analysis by a CNN.Trial registrationGerman Clinical Trial Register (DRKS) Identifier: DRKS00013570.

Project description:IntroductionAcute exacerbations of chronic obstructive pulmonary disease (AECOPD) play a pivotal role in the burden and progressive course of chronic obstructive pulmonary disease (COPD). As such, disease management is predominantly based on the prevention of these episodes of acute worsening of respiratory symptoms. However, to date, personalised prediction and early and accurate diagnosis of AECOPD remain unsuccessful. Therefore, the current study was designed to explore which frequently measured biomarkers can predict an AECOPD and/or respiratory infection in patients with COPD. Moreover, the study aims to increase our understanding of the heterogeneity of AECOPD as well as the role of microbial composition and hostmicrobiome interactions to elucidate new disease biology in COPD.Methods and analysisThe 'Early diagnostic BioMARKers in Exacerbations of COPD' study is an exploratory, prospective, longitudinal, single-centre, observational study with 8-week follow-up enrolling up to 150 patients with COPD admitted to inpatient pulmonary rehabilitation at Ciro (Horn, the Netherlands). Respiratory symptoms, vitals, spirometry and nasopharyngeal, venous blood, spontaneous sputum and stool samples will be frequently collected for exploratory biomarker analysis, longitudinal characterisation of AECOPD (ie, clinical, functional and microbial) and to identify host-microbiome interactions. Genomic sequencing will be performed to identify mutations associated with increased risk of AECOPD and microbial infections. Predictors of time-to-first AECOPD will be modelled using Cox proportional hazards' regression. Multiomic analyses will provide a novel integration tool to generate predictive models and testable hypotheses about disease causation and predictors of disease progression.Ethics and disseminationThis protocol was approved by the Medical Research Ethics Committees United (MEC-U), Nieuwegein, the Netherlands (NL71364.100.19).Trial registration numberNCT05315674.

Project description:BackgroundThe incidence of thyroid cancer is rising steadily because of overdiagnosis and overtreatment conferred by widespread use of sensitive imaging techniques for screening. This overall incidence growth is especially driven by increased diagnosis of indolent and well-differentiated papillary subtype and early-stage thyroid cancer, whereas the incidence of advanced-stage thyroid cancer has increased marginally. Thyroid ultrasound is frequently used to diagnose thyroid cancer. The aim of this study was to use deep convolutional neural network (DCNN) models to improve the diagnostic accuracy of thyroid cancer by analysing sonographic imaging data from clinical ultrasounds.MethodsWe did a retrospective, multicohort, diagnostic study using ultrasound images sets from three hospitals in China. We developed and trained the DCNN model on the training set, 131 731 ultrasound images from 17 627 patients with thyroid cancer and 180 668 images from 25 325 controls from the thyroid imaging database at Tianjin Cancer Hospital. Clinical diagnosis of the training set was made by 16 radiologists from Tianjin Cancer Hospital. Images from anatomical sites that were judged as not having cancer were excluded from the training set and only individuals with suspected thyroid cancer underwent pathological examination to confirm diagnosis. The model's diagnostic performance was validated in an internal validation set from Tianjin Cancer Hospital (8606 images from 1118 patients) and two external datasets in China (the Integrated Traditional Chinese and Western Medicine Hospital, Jilin, 741 images from 154 patients; and the Weihai Municipal Hospital, Shandong, 11 039 images from 1420 patients). All individuals with suspected thyroid cancer after clinical examination in the validation sets had pathological examination. We also compared the specificity and sensitivity of the DCNN model with the performance of six skilled thyroid ultrasound radiologists on the three validation sets.FindingsBetween Jan 1, 2012, and March 28, 2018, ultrasound images for the four study cohorts were obtained. The model achieved high performance in identifying thyroid cancer patients in the validation sets tested, with area under the curve values of 0·947 (95% CI 0·935-0·959) for the Tianjin internal validation set, 0·912 (95% CI 0·865-0·958) for the Jilin external validation set, and 0·908 (95% CI 0·891-0·925) for the Weihai external validation set. The DCNN model also showed improved performance in identifying thyroid cancer patients versus skilled radiologists. For the Tianjin internal validation set, sensitivity was 93·4% (95% CI 89·6-96·1) versus 96·9% (93·9-98·6; p=0·003) and specificity was 86·1% (81·1-90·2) versus 59·4% (53·0-65·6; p<0·0001). For the Jilin external validation set, sensitivity was 84·3% (95% CI 73·6-91·9) versus 92·9% (84·1-97·6; p=0·048) and specificity was 86·9% (95% CI 77·8-93·3) versus 57·1% (45·9-67·9; p<0·0001). For the Weihai external validation set, sensitivity was 84·7% (95% CI 77·0-90·7) versus 89·0% (81·9-94·0; p=0·25) and specificity was 87·8% (95% CI 81·6-92·5) versus 68·6% (60·7-75·8; p<0·0001).InterpretationThe DCNN model showed similar sensitivity and improved specificity in identifying patients with thyroid cancer compared with a group of skilled radiologists. The improved technical performance of the DCNN model warrants further investigation as part of randomised clinical trials.FundingThe Program for Changjiang Scholars and Innovative Research Team in University in China, and National Natural Science Foundation of China.