A Review of Multimodal Medical Image Fusion Techniques.
Ontology highlight
ABSTRACT:
SUBMITTER: Huang B
PROVIDER: S-EPMC7195632 | biostudies-literature | 2020
REPOSITORIES: biostudies-literature
Publications
A Review of Multimodal Medical Image Fusion Techniques.
Huang Bing B Yang Feng F Yin Mengxiao M Mo Xiaoying X Zhong Cheng C
Computational and mathematical methods in medicine 20200423
The medical image fusion is the process of coalescing multiple images from multiple imaging modalities to obtain a fused image with a large amount of information for increasing the clinical applicability of medical images. In this paper, we attempt to give an overview of multimodal medical image fusion methods, putting emphasis on the most recent advances in the domain based on (1) the current fusion methods, including based on deep learning, (2) imaging modalities of medical image fusion, and ( ...[more]
Similar Datasets
Project description:This study proposed an end-to-end unsupervised medical fusion generative adversarial network, MedFusionGAN, to fuse computed tomography (CT) and high-resolution isotropic 3D T1-Gd Magnetic resonance imaging (MRI) image sequences to generate an image with CT bone structure and MRI soft tissue contrast to improve target delineation and to reduce the radiotherapy planning time. We used a publicly available multicenter medical dataset (GLIS-RT, 230 patients) from the Cancer Imaging Archive. To improve the models generalization, we consider different imaging protocols and patients with various brain tumor types, including metastases. The proposed MedFusionGAN consisted of one generator network and one discriminator network trained in an adversarial scenario. Content, style, and L1 losses were used for training the generator to preserve the texture and structure information of the MRI and CT images. The MedFusionGAN successfully generates fused images with MRI soft-tissue and CT bone contrast. The results of the MedFusionGAN were quantitatively and qualitatively compared with seven traditional and eight deep learning (DL) state-of-the-art methods. Qualitatively, our method fused the source images with the highest spatial resolution without adding the image artifacts. We reported nine quantitative metrics to quantify the preservation of structural similarity, contrast, distortion level, and image edges in fused images. Our method outperformed both traditional and DL methods on six out of nine metrics. And it got the second performance rank for three and two quantitative metrics when compared with traditional and DL methods, respectively. To compare soft-tissue contrast, intensity profile along tumor and tumor contours of the fusion methods were evaluated. MedFusionGAN provides a more consistent, better intensity profile, and a better segmentation performance. The proposed end-to-end unsupervised method successfully fused MRI and CT images. The fused image could improve targets and OARs delineation, which is an important aspect of radiotherapy treatment planning.
| S-EPMC10704723 | biostudies-literature
Project description:Background contextFluoroscopic devices can be used to visualize subcutaneous and osseous tissue, a useful feature during pedicle screw insertion in lumbar fusion surgery. It is important that both patient and surgeon are exposed as little as possible, since these devices use potential harmful ionizing radiation.PurposeThis study aims to compare radiation exposure of different image-guided techniques in lumbar fusion surgery with pedicle screw insertion.Study designSystematic review.MethodsCochrane, Embase, PubMed and Web of Science databases were used to acquire relevant studies. Eligibility criteria were lumbar and/or sacral spine, pedicle screw, mGray and/or Sievert and/or mrem, radiation dose and/or radiation exposure. Image-guided techniques were divided in five groups: conventional C-arm, C-arm navigation, C-arm robotic, O-arm navigation and O-arm robotic. Comparisons were made based on effective dose for patients and surgeons, absorbed dose for patients and surgeons and exposure. Risk of bias was assessed using the 2017 Cochrane Risk of Bias tool on RCTs and the Cochrane ROBINS-I tool on NRCTs. Level of evidence was assessed using the guidelines of Oxford Centre for Evidence-based Medicine 2011.ResultsA total of 1423 studies were identified of which 38 were included in the analysis and assigned to one of the five groups. Results of radiation dose per procedure and per pedicle screw were described in dose ranges. Conventional C-arm appeared to result in higher effective dose for surgeons, higher absorbed dose for patients and higher exposure, compared to C-arm navigation/robotic and O-arm navigation/robotic. Level of evidence was 3 to 4 in 29 studies. Risk of bias of RCTs was intermediate, mostly due to inadequate blinding. Overall risk of bias score in NRCTs was determined as 'serious'.ConclusionsRanges of radiation doses using different modalities during pedicle screw insertion in lumbar fusion surgery are wide. Based on the highest numbers in the ranges, conventional C-arm tends to lead to a higher effective dose for surgeons, higher absorbed dose for patients and higher exposure, compared to C-arm-, and O-arm navigation/robotic. The level of evidence is low and risk of bias is fairly high. In future studies, heterogeneity should be limited by standardizing measurement methods and thoroughly describing the image-guided technique settings.
| S-EPMC9898805 | biostudies-literature
Project description:Alzheimer's disease (AD) is one of the most common neurodegenerative diseases. The formation of amyloid plaques by aggregated amyloid beta (A?) peptides is a primary event in AD pathology. Understanding the metabolomic features and related pathways is critical for studying plaque-related pathological events (e.g., cell death and neuron dysfunction). Mass spectrometry imaging (MSI), due to its high sensitivity and ability to obtain the spatial distribution of metabolites, has been applied to AD studies. However, limited studies of metabolites in amyloid plaques have been performed due to the drawbacks of the commonly used techniques such as matrix-assisted laser desorption/ionization MSI. In the current study, we obtained high spatial resolution (?17 ?m) MS images of the AD mouse brain using the Single-probe, a microscale sampling and ionization device, coupled to a mass spectrometer under ambient conditions. The adjacent slices were used to obtain fluorescence microscopy images to locate amyloid plaques. The MS image and the fluorescence microscopy image were fused to spatially correlate histological protein hallmarks with metabolomic features. The fused images produced significantly improved spatial resolution (?5 ?m), allowing for the determination of fine structures in MS images and metabolomic biomarkers representing amyloid plaques.
| S-EPMC6885010 | biostudies-literature
Project description:PurposeSparsity of annotated data is a major limitation in medical image processing tasks such as registration. Registered multimodal image data are essential for the diagnosis of medical conditions and the success of interventional medical procedures. To overcome the shortage of data, we present a method that allows the generation of annotated multimodal 4D datasets.MethodsWe use a CycleGAN network architecture to generate multimodal synthetic data from the 4D extended cardiac-torso (XCAT) phantom and real patient data. Organ masks are provided by the XCAT phantom; therefore, the generated dataset can serve as ground truth for image segmentation and registration. Realistic simulation of respiration and heartbeat is possible within the XCAT framework. To underline the usability as a registration ground truth, a proof of principle registration is performed.ResultsCompared to real patient data, the synthetic data showed good agreement regarding the image voxel intensity distribution and the noise characteristics. The generated T1-weighted magnetic resonance imaging, computed tomography (CT), and cone beam CT images are inherently co-registered. Thus, the synthetic dataset allowed us to optimize registration parameters of a multimodal non-rigid registration, utilizing liver organ masks for evaluation.ConclusionOur proposed framework provides not only annotated but also multimodal synthetic data which can serve as a ground truth for various tasks in medical imaging processing. We demonstrated the applicability of synthetic data for the development of multimodal medical image registration algorithms.
| S-EPMC8295129 | biostudies-literature
Project description:Medical image fusion techniques can fuse medical images from different morphologies to make the medical diagnosis more reliable and accurate, which play an increasingly important role in many clinical applications. To obtain a fused image with high visual quality and clear structure details, this paper proposes a convolutional neural network (CNN) based medical image fusion algorithm. The proposed algorithm uses the trained Siamese convolutional network to fuse the pixel activity information of source images to realize the generation of weight map. Meanwhile, a contrast pyramid is implemented to decompose the source image. According to different spatial frequency bands and a weighted fusion operator, source images are integrated. The results of comparative experiments show that the proposed fusion algorithm can effectively preserve the detailed structure information of source images and achieve good human visual effects.
| S-EPMC7218740 | biostudies-literature
Project description:Computed tomography (CT) images show structural features, while magnetic resonance imaging (MRI) images represent brain tissue anatomy but do not contain any functional information. How to effectively combine the images of the two modes has become a research challenge. In this paper, a new framework for medical image fusion is proposed which combines convolutional neural networks (CNNs) and non-subsampled shearlet transform (NSST) to simultaneously cover the advantages of them both. This method effectively retains the functional information of the CT image and reduces the loss of brain structure information and spatial distortion of the MRI image. In our fusion framework, the initial weights integrate the pixel activity information from two source images that is generated by a dual-branch convolutional network and is decomposed by NSST. Firstly, the NSST is performed on the source images and the initial weights to obtain their low-frequency and high-frequency coefficients. Then, the first component of the low-frequency coefficients is fused by a novel fusion strategy, which simultaneously copes with two key issues in the fusion processing which are named energy conservation and detail extraction. The second component of the low-frequency coefficients is fused by the strategy that is designed according to the spatial frequency of the weight map. Moreover, the high-frequency coefficients are fused by the high-frequency components of the initial weight. Finally, the final image is reconstructed by the inverse NSST. The effectiveness of the proposed method is verified using pairs of multimodality images, and the sufficient experiments indicate that our method performs well especially for medical image fusion.
| S-EPMC7178475 | biostudies-literature
Project description:Background and objectiveTransformers, which have been widely recognized as state-of-the-art tools in natural language processing (NLP), have also come to be recognized for their value in computer vision tasks. With this increasing popularity, they have also been extensively researched in the more complex medical imaging domain. The associated developments have resulted in transformers being on par with sought-after convolution neural networks, particularly for medical image segmentation. Methods combining both types of networks have proven to be especially successful in capturing local and global contexts, thereby significantly boosting their performances in various segmentation problems. Motivated by this success, we have attempted to survey the consequential research focused on innovative transformer networks, specifically those designed to cater to medical image segmentation in an efficient manner.MethodsDatabases like Google Scholar, arxiv, ResearchGate, Microsoft Academic, and Semantic Scholar have been utilized to find recent developments in this field. Specifically, research in the English language from 2021 to 2023 was considered.Key content and findingsIn this survey, we look into the different types of architectures and attention mechanisms that uniquely improve performance and the structures that are in place to handle complex medical data. Through this survey, we summarize the popular and unconventional transformer-based research as seen through different key angles and analyze quantitatively the strategies that have proven more advanced.ConclusionsWe have also attempted to discern existing gaps and challenges within current research, notably highlighting the deficiency of annotated medical data for precise deep learning model training. Furthermore, potential future directions for enhancing transformers' utility in healthcare are outlined, encompassing strategies such as transfer learning and exploiting foundation models for specialized medical image segmentation.
| S-EPMC10722011 | biostudies-literature
Project description:PurposeTo allow for comprehensive noninvasive diagnostics of coronary artery disease (CAD) by using three-dimensional (3D) image fusion of CT coronary angiography, CT-derived fractional flow reserve (CT FFR), whole-heart dynamic 3D cardiac MRI perfusion, and 3D cardiac MRI late gadolinium enhancement (LGE).Materials and methodsSeventeen patients (54 years ± 10 [standard deviation], one female) who underwent cardiac CT and cardiac MRI were included (combined subcohort of three prospective trials). Software facilitating multimodal 3D image fusion was developed. Postprocessing of CT data included segmentation of the coronary tree and heart contours, calculation of CT FFR values, and color coding of the coronary tree according to CT FFR. Postprocessing of cardiac MRI data included segmentation of the left ventricle (LV) in cardiac MRI perfusion and cardiac MRI LGE, co-registration of cardiac MRI to CT data, and projection of cardiac MRI perfusion and LGE values onto the high spatial resolution LV from CT.ResultsImage quality was rated as good to excellent (scores: 2.5-2.6; 3 = excellent). CT coronary angiography revealed significant stenoses in seven of 17 cases (41%). CT FFR was possible in 16 of 17 cases (94%) and showed pathologic flow in seven of 17 cases (41%), six of which coincided with cases revealing significant stenoses at CT coronary angiography. Cardiac MRI perfusion identified eight of 17 patients (47%) with hypoperfusion (ischemic burden of 17% ± 5). Cardiac MRI LGE showed myocardial scar in three of 17 cases (18%, scar burden of 7% ± 4). Conventional two-dimensional readout of CT coronary angiography and cardiac MRI resulted in eight of 17 cases (47%) with uncertain findings. Most of these divergent findings could be solved when adding information from CT FFR and 3D image fusion (six of eight, 75%).ConclusionMultimodal 3D cardiac image fusion is feasible and may help with comprehensive noninvasive CAD diagnostics.Supplemental material is available for this article.© RSNA, 2020.
| S-EPMC7977970 | biostudies-literature
Project description:Diseases pose a significant threat to the citrus industry, and the accurate detection of these diseases represent key factors for their early diagnosis and precise control. Existing diagnostic methods primarily rely on image models trained on vast datasets and limited their applicability due to singular backgrounds. To devise a more accurate, robust, and versatile model for citrus disease classification, this study focused on data diversity, knowledge assistance, and modal fusion. Leaves from healthy plants and plants infected with 10 prevalent diseases (citrus greening, citrus canker, anthracnose, scab, greasy spot, melanose, sooty mold, nitrogen deficiency, magnesium deficiency, and iron deficiency) were used as materials. Initially, three datasets with white, natural, and mixed backgrounds were constructed to analyze their effects on the training accuracy, test generalization ability, and classification balance. This diversification of data significantly improved the model's adaptability to natural settings. Subsequently, by leveraging agricultural domain knowledge, a structured citrus disease features glossary was developed to enhance the efficiency of data preparation and the credibility of identification results. To address the underutilization of multimodal data in existing models, this study explored semantic embedding methods for disease images and structured descriptive texts. Convolutional networks with different depths (VGG16, ResNet50, MobileNetV2, and ShuffleNetV2) were used to extract the visual features of leaves. Concurrently, TextCNN and fastText were used to extract textual features and semantic relationships. By integrating the complementary nature of the image and text information, a joint learning model for citrus disease features was achieved. ShuffleNetV2 + TextCNN, the optimal multimodal model, achieved a classification accuracy of 98.33% on the mixed dataset, which represented improvements of 9.78% and 21.11% over the single-image and single-text models, respectively. This model also exhibited faster convergence, superior classification balance, and enhanced generalization capability, compared with the other methods. The image-text multimodal feature fusion network proposed in this study, which integrates text and image features with domain knowledge, can identify and classify citrus diseases in scenarios with limited samples and multiple background noise. The proposed model provides a more reliable decision-making basis for the precise application of biological and chemical control strategies for citrus production.
| S-EPMC10711083 | biostudies-literature
Project description:BackgroundTransfer learning (TL) with convolutional neural networks aims to improve performances on a new task by leveraging the knowledge of similar tasks learned in advance. It has made a major contribution to medical image analysis as it overcomes the data scarcity problem as well as it saves time and hardware resources. However, transfer learning has been arbitrarily configured in the majority of studies. This review paper attempts to provide guidance for selecting a model and TL approaches for the medical image classification task.Methods425 peer-reviewed articles were retrieved from two databases, PubMed and Web of Science, published in English, up until December 31, 2020. Articles were assessed by two independent reviewers, with the aid of a third reviewer in the case of discrepancies. We followed the PRISMA guidelines for the paper selection and 121 studies were regarded as eligible for the scope of this review. We investigated articles focused on selecting backbone models and TL approaches including feature extractor, feature extractor hybrid, fine-tuning and fine-tuning from scratch.ResultsThe majority of studies (n = 57) empirically evaluated multiple models followed by deep models (n = 33) and shallow (n = 24) models. Inception, one of the deep models, was the most employed in literature (n = 26). With respect to the TL, the majority of studies (n = 46) empirically benchmarked multiple approaches to identify the optimal configuration. The rest of the studies applied only a single approach for which feature extractor (n = 38) and fine-tuning from scratch (n = 27) were the two most favored approaches. Only a few studies applied feature extractor hybrid (n = 7) and fine-tuning (n = 3) with pretrained models.ConclusionThe investigated studies demonstrated the efficacy of transfer learning despite the data scarcity. We encourage data scientists and practitioners to use deep models (e.g. ResNet or Inception) as feature extractors, which can save computational costs and time without degrading the predictive power.
| S-EPMC9007400 | biostudies-literature