Dataset Information

Predicting treatment response from longitudinal images using multi-task deep learning.

ABSTRACT: Radiographic imaging is routinely used to evaluate treatment response in solid tumors. Current imaging response metrics do not reliably predict the underlying biological response. Here, we present a multi-task deep learning approach that allows simultaneous tumor segmentation and response prediction. We design two Siamese subnetworks that are joined at multiple layers, which enables integration of multi-scale feature representations and in-depth comparison of pre-treatment and post-treatment images. The network is trained using 2568 magnetic resonance imaging scans of 321 rectal cancer patients for predicting pathologic complete response after neoadjuvant chemoradiotherapy. In multi-institution validation, the imaging-based model achieves AUC of 0.95 (95% confidence interval: 0.91-0.98) and 0.92 (0.87-0.96) in two independent cohorts of 160 and 141 patients, respectively. When combined with blood-based tumor markers, the integrated model further improves prediction accuracy with AUC 0.97 (0.93-0.99). Our approach to capturing dynamic information in longitudinal images may be broadly used for screening, treatment response evaluation, disease monitoring, and surveillance.

SUBMITTER: Jin C

PROVIDER: S-EPMC7994301 | biostudies-literature | 2021 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Predicting treatment response from longitudinal images using multi-task deep learning.

Jin Cheng C Yu Heng H Ke Jia J Ding Peirong P Yi Yongju Y Jiang Xiaofeng X Duan Xin X Tang Jinghua J Chang Daniel T DT Wu Xiaojian X Gao Feng F Li Ruijiang R

Nature communications 20210325 1

Radiographic imaging is routinely used to evaluate treatment response in solid tumors. Current imaging response metrics do not reliably predict the underlying biological response. Here, we present a multi-task deep learning approach that allows simultaneous tumor segmentation and response prediction. We design two Siamese subnetworks that are joined at multiple layers, which enables integration of multi-scale feature representations and in-depth comparison of pre-treatment and post-treatment ima ...[more]

PMID: 33767170

Similar Datasets

Project description:ObjectiveEarly predicting response before neoadjuvant chemotherapy (NAC) is crucial for personalized treatment plans for locally advanced breast cancer patients. We aim to develop a multi-task model using multiscale whole slide images (WSIs) features to predict the response to breast cancer NAC more finely.MethodsThis work collected 1,670 whole slide images for training and validation sets, internal testing sets, external testing sets, and prospective testing sets of the weakly-supervised deep learning-based multi-task model (DLMM) in predicting treatment response and pCR to NAC. Our approach models two-by-two feature interactions across scales by employing concatenate fusion of single-scale feature representations, and controls the expressiveness of each representation via a gating-based attention mechanism.ResultsIn the retrospective analysis, DLMM exhibited excellent predictive performance for the prediction of treatment response, with area under the receiver operating characteristic curves (AUCs) of 0.869 [95% confidence interval (95% CI): 0.806-0.933] in the internal testing set and 0.841 (95% CI: 0.814-0.867) in the external testing sets. For the pCR prediction task, DLMM reached AUCs of 0.865 (95% CI: 0.763-0.964) in the internal testing and 0.821 (95% CI: 0.763-0.878) in the pooled external testing set. In the prospective testing study, DLMM also demonstrated favorable predictive performance, with AUCs of 0.829 (95% CI: 0.754-0.903) and 0.821 (95% CI: 0.692-0.949) in treatment response and pCR prediction, respectively. DLMM significantly outperformed the baseline models in all testing sets (P<0.05). Heatmaps were employed to interpret the decision-making basis of the model. Furthermore, it was discovered that high DLMM scores were associated with immune-related pathways and cells in the microenvironment during biological basis exploration.ConclusionsThe DLMM represents a valuable tool that aids clinicians in selecting personalized treatment strategies for breast cancer patients.

Project description:Background18 F-fluorodeoxyglucose (FDG) positron emission tomography (PET) is valuable for determining presence of viable tumor, but is limited by geographical restrictions, radiation exposure, and high cost.PurposeTo generate diagnostic-quality PET equivalent imaging for patients with brain neoplasms by deep learning with multi-contrast MRI.Study typeRetrospective.SubjectsPatients (59 studies from 51 subjects; age 56 ± 13 years; 29 males) who underwent 18 F-FDG PET and MRI for determining recurrent brain tumor.Field strength/sequence3T; 3D GRE T1, 3D GRE T1c, 3D FSE T2-FLAIR, and 3D FSE ASL, 18 F-FDG PET imaging.AssessmentConvolutional neural networks were trained using four MRIs as inputs and acquired FDG PET images as output. The agreement between the acquired and synthesized PET was evaluated by quality metrics and Bland-Altman plots for standardized uptake value ratio. Three physicians scored image quality on a 5-point scale, with score ≥3 as high-quality. They assessed the lesions on a 5-point scale, which was binarized to analyze diagnostic consistency of the synthesized PET compared to the acquired PET.Statistical testsThe agreement in ratings between the acquired and synthesized PET were tested with Gwet's AC and exact Bowker test of symmetry. Agreement of the readers was assessed by Gwet's AC. P = 0.05 was used as the cutoff for statistical significance.ResultsThe synthesized PET visually resembled the acquired PET and showed significant improvement in quality metrics (+21.7% on PSNR, +22.2% on SSIM, -31.8% on RSME) compared with ASL. A total of 49.7% of the synthesized PET were considered as high-quality compared to 73.4% of the acquired PET which was statistically significant, but with distinct variability between readers. For the positive/negative lesion assessment, the synthesized PET had an accuracy of 87% but had a tendency to overcall.ConclusionThe proposed deep learning model has the potential of synthesizing diagnostic quality FDG PET images without the use of radiotracers.Evidence level3 TECHNICAL EFFICACY: Stage 2.

Project description:The COVID-19 pandemic is causing a major outbreak in more than 150 countries around the world, having a severe impact on the health and life of many people globally. One of the crucial step in fighting COVID-19 is the ability to detect the infected patients early enough, and put them under special care. Detecting this disease from radiography and radiology images is perhaps one of the fastest ways to diagnose the patients. Some of the early studies showed specific abnormalities in the chest radiograms of patients infected with COVID-19. Inspired by earlier works, we study the application of deep learning models to detect COVID-19 patients from their chest radiography images. We first prepare a dataset of 5000 Chest X-rays from the publicly available datasets. Images exhibiting COVID-19 disease presence were identified by board-certified radiologist. Transfer learning on a subset of 2000 radiograms was used to train four popular convolutional neural networks, including ResNet18, ResNet50, SqueezeNet, and DenseNet-121, to identify COVID-19 disease in the analyzed chest X-ray images. We evaluated these models on the remaining 3000 images, and most of these networks achieved a sensitivity rate of 98% ( ± 3%), while having a specificity rate of around 90%. Besides sensitivity and specificity rates, we also present the receiver operating characteristic (ROC) curve, precision-recall curve, average prediction, and confusion matrix of each model. We also used a technique to generate heatmaps of lung regions potentially infected by COVID-19 and show that the generated heatmaps contain most of the infected areas annotated by our board certified radiologist. While the achieved performance is very encouraging, further analysis is required on a larger set of COVID-19 images, to have a more reliable estimation of accuracy rates. The dataset, model implementations (in PyTorch), and evaluations, are all made publicly available for research community at https://github.com/shervinmin/DeepCovid.git.

Dataset Information

Predicting treatment response from longitudinal images using multi-task deep learning.

Publications

Predicting treatment response from longitudinal images using multi-task deep learning.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets