Dataset Information

Interpretable multimodal deep learning for real-time pan-tissue pan-disease pathology search on social media.

ABSTRACT: Pathologists are responsible for rapidly providing a diagnosis on critical health issues. Challenging cases benefit from additional opinions of pathologist colleagues. In addition to on-site colleagues, there is an active worldwide community of pathologists on social media for complementary opinions. Such access to pathologists worldwide has the capacity to improve diagnostic accuracy and generate broader consensus on next steps in patient care. From Twitter we curate 13,626 images from 6,351 tweets from 25 pathologists from 13 countries. We supplement the Twitter data with 113,161 images from 1,074,484 PubMed articles. We develop machine learning and deep learning models to (i) accurately identify histopathology stains, (ii) discriminate between tissues, and (iii) differentiate disease states. Area Under Receiver Operating Characteristic (AUROC) is 0.805-0.996 for these tasks. We repurpose the disease classifier to search for similar disease states given an image and clinical covariates. We report precision@k?=?1?=?0.7618?±?0.0018 (chance 0.397?±?0.004, mean?±stdev?). The classifiers find that texture and tissue are important clinico-visual features of disease. Deep features trained only on natural images (e.g., cats and dogs) substantially improved search performance, while pathology-specific deep features and cell nuclei features further improved search to a lesser extent. We implement a social media bot (@pathobot on Twitter) to use the trained classifiers to aid pathologists in obtaining real-time feedback on challenging cases. If a social media post containing pathology text and images mentions the bot, the bot generates quantitative predictions of disease state (normal/artifact/infection/injury/nontumor, preneoplastic/benign/low-grade-malignant-potential, or malignant) and lists similar cases across social media and PubMed. Our project has become a globally distributed expert system that facilitates pathological diagnosis and brings expertise to underserved regions or hospitals with less expertise in a particular disease. This is the first pan-tissue pan-disease (i.e., from infection to malignancy) method for prediction and search on social media, and the first pathology study prospectively tested in public on social media. We will share data through http://pathobotology.org . We expect our project to cultivate a more connected world of physicians and improve patient care worldwide.

SUBMITTER: Schaumberg AJ

PROVIDER: S-EPMC7581495 | biostudies-literature | 2020 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Interpretable multimodal deep learning for real-time pan-tissue pan-disease pathology search on social media.

Schaumberg Andrew J AJ Juarez-Nicanor Wendy C WC Choudhury Sarah J SJ Pastrián Laura G LG Pritt Bobbi S BS Prieto Pozuelo Mario M Sotillo Sánchez Ricardo R Ho Khanh K Zahra Nusrat N Sener Betul Duygu BD Yip Stephen S Xu Bin B Annavarapu Srinivas Rao SR Morini Aurélien A Jones Karra A KA Rosado-Orozco Kathia K Mukhopadhyay Sanjay S Miguel Carlos C Yang Hongyu H Rosen Yale Y Ali Rola H RH Folaranmi Olaleke O OO Gardner Jerad M JM Rusu Corina C Stayerman Celina C Gross John J Suleiman Dauda E DE Sirintrapun S Joseph SJ Aly Mariam M Fuchs Thomas J TJ

Modern pathology : an official journal of the United States and Canadian Academy of Pathology, Inc 20200528 11

Pathologists are responsible for rapidly providing a diagnosis on critical health issues. Challenging cases benefit from additional opinions of pathologist colleagues. In addition to on-site colleagues, there is an active worldwide community of pathologists on social media for complementary opinions. Such access to pathologists worldwide has the capacity to improve diagnostic accuracy and generate broader consensus on next steps in patient care. From Twitter we curate 13,626 images from 6,351 tw ...[more]

PMID: 32467650

Similar Datasets

Project description:MotivationIntegrating multimodal data represents an effective approach to predicting biomedical characteristics, such as protein functions and disease outcomes. However, existing data integration approaches do not sufficiently address the heterogeneous semantics of multimodal data. In particular, early and intermediate approaches that rely on a uniform integrated representation reinforce the consensus among the modalities but may lose exclusive local information. The alternative late integration approach that can address this challenge has not been systematically studied for biomedical problems.ResultsWe propose Ensemble Integration (EI) as a novel systematic implementation of the late integration approach. EI infers local predictive models from the individual data modalities using appropriate algorithms and uses heterogeneous ensemble algorithms to integrate these local models into a global predictive model. We also propose a novel interpretation method for EI models. We tested EI on the problems of predicting protein function from multimodal STRING data and mortality due to coronavirus disease 2019 (COVID-19) from multimodal data in electronic health records. We found that EI accomplished its goal of producing significantly more accurate predictions than each individual modality. It also performed better than several established early integration methods for each of these problems. The interpretation of a representative EI model for COVID-19 mortality prediction identified several disease-relevant features, such as laboratory test (blood urea nitrogen and calcium) and vital sign measurements (minimum oxygen saturation) and demographics (age). These results demonstrated the effectiveness of the EI framework for biomedical data integration and predictive modeling.Availability and implementationCode and data are available at https://github.com/GauravPandeyLab/ensemble_integration.Supplementary informationSupplementary data are available at Bioinformatics Advances online.

Dataset Information

Interpretable multimodal deep learning for real-time pan-tissue pan-disease pathology search on social media.

Publications

Interpretable multimodal deep learning for real-time pan-tissue pan-disease pathology search on social media.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets