Dataset Information

Estimating diagnostic uncertainty in artificial intelligence assisted pathology using conformal prediction.

ABSTRACT: Unreliable predictions can occur when an artificial intelligence (AI) system is presented with data it has not been exposed to during training. We demonstrate the use of conformal prediction to detect unreliable predictions, using histopathological diagnosis and grading of prostate biopsies as example. We digitized 7788 prostate biopsies from 1192 men in the STHLM3 diagnostic study, used for training, and 3059 biopsies from 676 men used for testing. With conformal prediction, 1 in 794 (0.1%) predictions is incorrect for cancer diagnosis (compared to 14 errors [2%] without conformal prediction) while 175 (22%) of the predictions are flagged as unreliable when the AI-system is presented with new data from the same lab and scanner that it was trained on. Conformal prediction could with small samples (N = 49 for external scanner, N = 10 for external lab and scanner, and N = 12 for external lab, scanner and pathology assessment) detect systematic differences in external data leading to worse predictive performance. The AI-system with conformal prediction commits 3 (2%) errors for cancer detection in cases of atypical prostate tissue compared to 44 (25%) without conformal prediction, while the system flags 143 (80%) unreliable predictions. We conclude that conformal prediction can increase patient safety of AI-systems.

SUBMITTER: Olsson H

PROVIDER: S-EPMC9755280 | biostudies-literature | 2022 Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Estimating diagnostic uncertainty in artificial intelligence assisted pathology using conformal prediction.

Olsson Henrik H Kartasalo Kimmo K Mulliqi Nita N Capuccini Marco M Ruusuvuori Pekka P Samaratunga Hemamali H Delahunt Brett B Lindskog Cecilia C Janssen Emiel A M EAM Blilie Anders A Egevad Lars L Spjuth Ola O Eklund Martin M

Nature communications 20221215 1

Unreliable predictions can occur when an artificial intelligence (AI) system is presented with data it has not been exposed to during training. We demonstrate the use of conformal prediction to detect unreliable predictions, using histopathological diagnosis and grading of prostate biopsies as example. We digitized 7788 prostate biopsies from 1192 men in the STHLM3 diagnostic study, used for training, and 3059 biopsies from 676 men used for testing. With conformal prediction, 1 in 794 (0.1%) pre ...[more]

PMID: 36522311

Similar Datasets

Project description:The rising global incidence of cervical cancer is estimated to have affected more than 600,000 women, and nearly 350,000 women are predicted to have died from the disease in 2020 alone. Novel advances in cancer prevention, screening, diagnosis and treatment have all but reduced the burden of cervical cancer in developed nations. Unfortunately, cervical cancer is still the number one gynecological cancer globally. A limiting factor in managing cervical cancer globally is access to healthcare systems and trained medical personnel. Any methodology or procedure that may simplify or assist cervical cancer screening is desirable. Herein, we assess the use of artificial intelligence (AI)-assisted colposcopy in a tertiary hospital cervical diagnostic pathology unit. The study group consisted of 48 women (mean age 34) who were referred to the clinic for a routine colposcopy by their gynecologist. Cervical images were taken by an EVA-Visualcheck TM colposcope and run through an AI algorithm that gave real-time binary results of the cervical images as being either normal or abnormal. The primary endpoint of the study assessed the AI algorithm's ability to correctly identify histopathology results of CIN2+ as being abnormal. A secondary endpoint was a comparison between the AI algorithm and the clinical assessment results. Overall, we saw lower sensitivity of AI (66.7%; 12/18) compared with the clinical assessment (100%; 18/18), and histopathology results as the gold standard. The positive predictive value (PPV) was comparable between AI (42.9%; 12/28) and the clinical assessment (41.8%; 18/43). The specificity, however, was higher in the AI algorithm (46.7%; 14/30) compared to the clinical assessment (16.7%; 5/30). Comparing the congruence between the AI algorithm and histopathology results showed agreement 54.2% of the time and disagreement 45.8% of the time. A trained colposcopist was in agreement 47.9% and disagreement 52.1% of the time. Assessing these results, there is currently no added benefit of using the AI algorithm as a tool of speeding up diagnosis. However, given the steady improvements in the AI field, we believe that AI-assisted colposcopy may be of use in the future.

Project description:Diffuse large B-cell lymphoma (DLBCL) is a heterogeneous and prevalent subtype of aggressive non-Hodgkin lymphoma that poses diagnostic and prognostic challenges, particularly in predicting drug responsiveness. In this study, we used digital pathology and deep learning to predict responses to immunochemotherapy in patients with DLBCL. We retrospectively collected 251 slide images from 216 DLBCL patients treated with rituximab, cyclophosphamide, doxorubicin, vincristine, and prednisone (R-CHOP), with their immunochemotherapy response labels. The digital pathology images were processed using contrastive learning for feature extraction. A multi-modal prediction model was developed by integrating clinical data and pathology image features. Knowledge distillation was employed to mitigate overfitting on gigapixel histopathology images to create a model that predicts responses based solely on pathology images. Based on the importance derived from the attention mechanism of the model, we extracted histological features that were considered key textures associated with drug responsiveness. The multi-modal prediction model achieved an impressive area under the ROC curve of 0.856, demonstrating significant associations with clinical variables such as Ann Arbor stage, International Prognostic Index, and bulky disease. Survival analyses indicated their effectiveness in predicting relapse-free survival. External validation using TCGA datasets supported the model's ability to predict survival differences. Additionally, pathology-based predictions show promise as independent prognostic indicators. Histopathological analysis identified centroblastic and immunoblastic features to be associated with treatment response, aligning with previous morphological classifications and highlighting the objectivity and reproducibility of artificial intelligence-based diagnosis. This study introduces a novel approach that combines digital pathology and clinical data to predict the response to immunochemotherapy in patients with DLBCL. This model shows great promise as a diagnostic and prognostic tool for clinical management of DLBCL. Further research and genomic data integration hold the potential to enhance its impact on clinical practice, ultimately improving patient outcomes.

Project description:The diagnosis of Parkinson's disease (PD) is challenging at all stages due to variable symptomatology, comorbidities, and mimicking conditions. Postmortem assessment remains the gold standard for a definitive diagnosis. While it is well recognized that PD manifests pathologically in the central nervous system with aggregation of α-synuclein as Lewy bodies and neurites, similar Lewy-type synucleinopathy (LTS) is additionally found in the peripheral nervous system that may be useful as an antemortem biomarker. We have previously found that detection of LTS in submandibular gland (SMG) biopsies is sensitive and specific for advanced PD; however, the sensitivity is suboptimal especially for early-stage disease. Further, visual microscopic assessment of biopsies by a neuropathologist to identify LTS is impractical for large-scale adoption. Here, we trained and validated a convolutional neural network (CNN) for detection of LTS on 283 digital whole slide images (WSI) from 95 unique SMG biopsies. A total of 8,450 LTS and 35,066 background objects were annotated following an inter-rater reliability study with Fleiss Kappa = 0.72. We used transfer learning to train a CNN model to classify image patches (151 × 151 pixels at 20× magnification) with and without the presence of LTS objects. The trained CNN model showed the following performance on image patches: sensitivity: 0.99, specificity: 0.99, precision: 0.81, accuracy: 0.99, and F-1 score: 0.89. We further tested the trained network on 1230 naïve WSI from the same cohort of research subjects comprising 42 PD patients and 14 controls. Logistic regression models trained on features engineered from the CNN predictions on the WSI resulted in sensitivity: 0.71, specificity: 0.65, precision: 0.86, accuracy: 0.69, and F-1 score: 0.76 in predicting clinical PD status, and 0.64 accuracy in predicting PD stage, outperforming expert neuropathologist LTS density scoring in terms of sensitivity but not specificity. These findings demonstrate the practical utility of a CNN detector in screening for LTS, which can translate into a computational tool to facilitate the antemortem tissue-based diagnosis of PD in clinical settings.

Dataset Information

Estimating diagnostic uncertainty in artificial intelligence assisted pathology using conformal prediction.

Publications

Estimating diagnostic uncertainty in artificial intelligence assisted pathology using conformal prediction.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets