Multimodal Representation Learning via Maximization of Local Mutual Information.
Ontology highlight
ABSTRACT:
SUBMITTER: Liao R
PROVIDER: S-EPMC9576150 | biostudies-literature | 2021 Sep-Oct
REPOSITORIES: biostudies-literature
Publications
Multimodal Representation Learning via Maximization of Local Mutual Information.
Liao Ruizhi R Moyer Daniel D Cha Miriam M Quigley Keegan K Berkowitz Seth S Horng Steven S Golland Polina P Wells William M WM
Medical image computing and computer-assisted intervention : MICCAI ... International Conference on Medical Image Computing and Computer-Assisted Intervention 20210921
We propose and demonstrate a representation learning approach by maximizing the mutual information between local features of images and text. The goal of this approach is to learn <i>useful</i> image representations by taking advantage of the rich information contained in the free text that describes the findings in the image. Our method trains image and text encoders by encouraging the resulting representations to exhibit high local mutual information. We make use of recent advances in mutual i ...[more]
Similar Datasets
Project description:Applying network science approaches to investigate the functions and anatomy of the human brain is prevalent in modern medical imaging analysis. Due to the complex network topology, for an individual brain, mining a discriminative network representation from the multimodal brain networks is non-trivial. The recent success of deep learning techniques on graph-structured data suggests a new way to model the non-linear cross-modality relationship. However, current deep brain network methods either ignore the intrinsic graph topology or require a network basis shared within a group. To address these challenges, we propose a novel end-to-end deep graph representation learning (Deep Multimodal Brain Networks - DMBN) to fuse multimodal brain networks. Specifically, we decipher the cross-modality relationship through a graph encoding and decoding process. The higher-order network mappings from brain structural networks to functional networks are learned in the node domain. The learned network representation is a set of node features that are informative to induce brain saliency maps in a supervised manner. We test our framework in both synthetic and real image data. The experimental results show the superiority of the proposed method over some other state-of-the-art deep brain network models.
| S-EPMC8293685 | biostudies-literature
Project description:Here we demonstrate the suitability of a local mutual information measure for estimating the temporal dynamics of cross-frequency coupling (CFC) in brain electrophysiological signals. In CFC, concurrent activity streams in different frequency ranges interact and transiently couple. A particular form of CFC, phase-amplitude coupling (PAC), has raised interest given the growing amount of evidence of its possible role in healthy and pathological brain information processing. Although several methods have been proposed for PAC estimation, only a few have addressed the estimation of the temporal evolution of PAC, and these typically require a large number of experimental trials to return a reliable estimate. Here we explore the use of mutual information to estimate a PAC measure (MIPAC) in both continuous and event-related multi-trial data. To validate these two applications of the proposed method, we first apply it to a set of simulated phase-amplitude modulated signals and show that MIPAC can successfully recover the temporal dynamics of the simulated coupling in either continuous or multi-trial data. Finally, to explore the use of MIPAC to analyze data from human event-related paradigms, we apply it to an actual event-related human electrocorticographic (ECoG) data set that exhibits strong PAC, demonstrating that the MIPAC estimator can be used to successfully characterize amplitude-modulation dynamics in electrophysiological data.
| S-EPMC6342492 | biostudies-literature
Project description:Pancreatic cancer is one of the most adverse diseases and it is very difficult to treat because the cancer cells formed in the pancreas intertwine themselves with nearby blood vessels and connective tissue. Hence, the surgical procedure of treatment becomes complicated and it does not always lead to a cure. Histopathological diagnosis is the usual approach for cancer diagnosis. However, the pancreas remains so deep inside the body that experts sometimes struggle to detect cancer in it. Computer-aided diagnosis can come to the aid of pathologists in this scenario. It assists experts by supporting their diagnostic decisions. In this research, we carried out a deep learning-based approach to analyze histopathology images. We collected whole-slide images of KPC mice to implement this work. The pancreatic abnormalities observed in KPC mice develop similar histological features to human beings. We created random patches from whole-slide images. Then, a convolutional autoencoder framework was used to embed these patches into an integrated latent space. We applied 'information maximization', a deep learning clustering technique to cluster the identical patches in an unsupervised manner since our dataset does not have annotation. Moreover, Uniform manifold approximation and projection, a nonlinear dimension reduction technique was utilized to visualize the embedded patches in a 2-dimensional space. Finally, we calculated a few internal cluster validation metrics to determine the optimal cluster set. Our work concentrated on patch-based anomaly detection in the whole slide histopathology images of KPC mice.
| S-EPMC10707605 | biostudies-literature
Project description:MotivationPredicting molecule-disease indications and side effects is important for drug development and pharmacovigilance. Comprehensively mining molecule-molecule, molecule-disease and disease-disease semantic dependencies can potentially improve prediction performance.MethodsWe introduce a Multi-Modal REpresentation Mapping Approach to Predicting molecular-disease relations (M2REMAP) by incorporating clinical semantics learned from electronic health records (EHR) of 12.6 million patients. Specifically, M2REMAP first learns a multimodal molecule representation that synthesizes chemical property and clinical semantic information by mapping molecule chemicals via a deep neural network onto the clinical semantic embedding space shared by drugs, diseases and other common clinical concepts. To infer molecule-disease relations, M2REMAP combines multimodal molecule representation and disease semantic embedding to jointly infer indications and side effects.ResultsWe extensively evaluate M2REMAP on molecule indications, side effects and interactions. Results show that incorporating EHR embeddings improves performance significantly, for example, attaining an improvement over the baseline models by 23.6% in PRC-AUC on indications and 23.9% on side effects. Further, M2REMAP overcomes the limitation of existing methods and effectively predicts drugs for novel diseases and emerging pathogens.Availability and implementationThe code is available at https://github.com/celehs/M2REMAP, and prediction results are provided at https://shiny.parse-health.org/drugs-diseases-dev/.Supplementary informationSupplementary data are available at Bioinformatics online.
| S-EPMC9940625 | biostudies-literature
Project description:MotivationAccurately predicting cancer survival is crucial for helping clinicians to plan appropriate treatments, which largely improves the life quality of cancer patients and spares the related medical costs. Recent advances in survival prediction methods suggest that integrating complementary information from different modalities, e.g. histopathological images and genomic data, plays a key role in enhancing predictive performance. Despite promising results obtained by existing multimodal methods, the disparate and heterogeneous characteristics of multimodal data cause the so-called modality gap problem, which brings in dramatically diverse modality representations in feature space. Consequently, detrimental modality gaps make it difficult for comprehensive integration of multimodal information via representation learning and therefore pose a great challenge to further improvements of cancer survival prediction.ResultsTo solve the above problems, we propose a novel method called cross-aligned multimodal representation learning (CAMR), which generates both modality-invariant and -specific representations for more accurate cancer survival prediction. Specifically, a cross-modality representation alignment learning network is introduced to reduce modality gaps by effectively learning modality-invariant representations in a common subspace, which is achieved by aligning the distributions of different modality representations through adversarial training. Besides, we adopt a cross-modality fusion module to fuse modality-invariant representations into a unified cross-modality representation for each patient. Meanwhile, CAMR learns modality-specific representations which complement modality-invariant representations and therefore provides a holistic view of the multimodal data for cancer survival prediction. Comprehensive experiment results demonstrate that CAMR can successfully narrow modality gaps and consistently yields better performance than other survival prediction methods using multimodal data.Availability and implementationCAMR is freely available at https://github.com/wxq-ustc/CAMR.Supplementary informationSupplementary data are available at Bioinformatics online.
| S-EPMC9857974 | biostudies-literature
Project description:MotivationDeep learning approaches have empowered single-cell omics data analysis in many ways and generated new insights from complex cellular systems. As there is an increasing need for single-cell omics data to be integrated across sources, types and features of data, the challenges of integrating single-cell omics data are rising. Here, we present an unsupervised deep learning algorithm that learns discriminative representations for single-cell data via maximizing mutual information, SMILE (Single-cell Mutual Information Learning).ResultsUsing a unique cell-pairing design, SMILE successfully integrates multisource single-cell transcriptome data, removing batch effects and projecting similar cell types, even from different tissues, into the shared space. SMILE can also integrate data from two or more modalities, such as joint-profiling technologies using single-cell ATAC-seq, RNA-seq, DNA methylation, Hi-C and ChIP data. When paired cells are known, SMILE can integrate data with unmatched feature, such as genes for RNA-seq and genome-wide peaks for ATAC-seq. Integrated representations learned from joint-profiling technologies can then be used as a framework for comparing independent single source data.Availability and implementationThe source code of SMILE including analyses of key results in the study can be found at: https://github.com/rpmccordlab/SMILE, implemented in Python.Supplementary informationSupplementary data are available at Bioinformatics online.
| S-EPMC10060712 | biostudies-literature
Project description:BackgroundThe definition of a distance measure plays a key role in the evaluation of different clustering solutions of gene expression profiles. In this empirical study we compare different clustering solutions when using the Mutual Information (MI) measure versus the use of the well known Euclidean distance and Pearson correlation coefficient.ResultsRelying on several public gene expression datasets, we evaluate the homogeneity and separation scores of different clustering solutions. It was found that the use of the MI measure yields a more significant differentiation among erroneous clustering solutions. The proposed measure was also used to analyze the performance of several known clustering algorithms. A comparative study of these algorithms reveals that their "best solutions" are ranked almost oppositely when using different distance measures, despite the found correspondence between these measures when analysing the averaged scores of groups of solutions.ConclusionIn view of the results, further attention should be paid to the selection of a proper distance measure for analyzing the clustering of gene expression data.
| S-EPMC1858704 | biostudies-literature
Project description:Over the past few years, online learning has exploded in popularity due to the potentially unlimited enrollment, lack of geographical limitations, and free accessibility of many courses. However, learners are prone to have poor performance due to the unconstrained learning environment, lack of academic pressure, and low interactivity. Personalized intervention design with the learners’ background and learning behavior factors in mind may improve the learners’ performance. Causality strictly distinguishes cause from outcome factors and plays an irreplaceable role in designing guiding interventions. The goal of this paper is to construct a Bayesian network to make causal analysis and then provide personalized interventions for different learners to improve learning. This paper first constructs a Bayesian network based on background and learning behavior factors, combining expert knowledge and a structure learning algorithm. Then the important factors in the constructed network are selected using mutual information based on entropy. At last, we identify learners with poor performance using inference and propose personalized interventions, which may help with successful applications in education. Experimental results verify the effectiveness of the proposed method and demonstrate the impact of factors on learning performance.
| S-EPMC7514446 | biostudies-literature
Project description:The rationality assumption that underlies mainstream economic theory has proved to be a useful approximation, despite the fact that systematic violations to its predictions can be found. That is, the assumption of rational behavior is useful in understanding the ways in which many successful economic institutions function, although it is also true that actual human behavior falls systematically short of perfect rationality. We consider a possible explanation of this apparent inconsistency, suggesting that mechanisms that rest on the rationality assumption are likely to be successful when they create an environment in which the behavior they try to facilitate leads to the best payoff for all agents on average, and most of the time. Review of basic learning research suggests that, under these conditions, people quickly learn to maximize expected return. This review also shows that there are many situations in which experience does not increase maximization. In many cases, experience leads people to underweight rare events. In addition, the current paper suggests that it is convenient to distinguish between two behavioral approaches to improve economic analyses. The first, and more conventional approach among behavioral economists and psychologists interested in judgment and decision making, highlights violations of the rational model and proposes descriptive models that capture these violations. The second approach studies human learning to clarify the conditions under which people quickly learn to maximize expected return. The current review highlights one set of conditions of this type and shows how the understanding of these conditions can facilitate market design.
| S-EPMC4113920 | biostudies-other
Project description:Abstract Motivation Use of multi-omics data carrying comprehensive signals about the disease is strongly desirable for understanding and predicting disease progression, cancer particularly as a serious disease with a high mortality rate. However, recent methods currently fail to effectively utilize the multi-omics data for cancer survival prediction and thus significantly limiting the accuracy of survival prediction using omics data. Results In this work, we constructed a deep learning model with multimodal representation and integration to predict the survival of patients using multi-omics data. We first developed an unsupervised learning part to extract high-level feature representations from omics data of different modalities. Then, we used an attention-based method to integrate feature representations, produced by the unsupervised learning part, into a single compact vector and finally we fed the vector into fully connected layers for survival prediction. We used multimodal data to train the model and predict pancancer survival, and the results show that using multimodal data can lead to higher prediction accuracy compared to using single modal data. Furthermore, we used the concordance index and the 5-fold cross-validation method for comparing our proposed method with current state-of-the-art methods and our results show that our model achieves better performance on the majority of cancer types in our testing datasets. Availability and implementation https://github.com/ZhangqiJiang07/MultimodalSurvivalPrediction. Supplementary information Supplementary data are available at Bioinformatics online.
| S-EPMC9945067 | biostudies-literature