Dataset Information

Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision.

ABSTRACT: Convolutional neural network (CNN) driven by image recognition has been shown to be able to explain cortical responses to static pictures at ventral-stream areas. Here, we further showed that such CNN could reliably predict and decode functional magnetic resonance imaging data from humans watching natural movies, despite its lack of any mechanism to account for temporal dynamics or feedback processing. Using separate data, encoding and decoding models were developed and evaluated for describing the bi-directional relationships between the CNN and the brain. Through the encoding models, the CNN-predicted areas covered not only the ventral stream, but also the dorsal stream, albeit to a lesser degree; single-voxel response was visualized as the specific pixel pattern that drove the response, revealing the distinct representation of individual cortical location; cortical activation was synthesized from natural images with high-throughput to map category representation, contrast, and selectivity. Through the decoding models, fMRI signals were directly decoded to estimate the feature representations in both visual and semantic spaces, for direct visual reconstruction and semantic categorization, respectively. These results corroborate, generalize, and extend previous findings, and highlight the value of using deep learning, as an all-in-one model of the visual cortex, to understand and decode natural vision.

SUBMITTER: Wen H

PROVIDER: S-EPMC6215471 | biostudies-literature | 2018 Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision.

Wen Haiguang H Shi Junxing J Zhang Yizhen Y Lu Kun-Han KH Cao Jiayue J Liu Zhongming Z

Cerebral cortex (New York, N.Y. : 1991) 20181201 12

Convolutional neural network (CNN) driven by image recognition has been shown to be able to explain cortical responses to static pictures at ventral-stream areas. Here, we further showed that such CNN could reliably predict and decode functional magnetic resonance imaging data from humans watching natural movies, despite its lack of any mechanism to account for temporal dynamics or feedback processing. Using separate data, encoding and decoding models were developed and evaluated for describing ...[more]

PMID: 29059288

Similar Datasets

Project description:Cell signaling dynamics and transcriptional regulatory activities are variable within specific cell types responding to an identical stimulus. In addition to studying the network interactions, there is much interest in utilizing single cell scale data to elucidate the non-random aspects of the variability involved in cellular decision making. Previous studies have considered the information transfer between the signaling and transcriptional domains based on an instantaneous relationship between the molecular activities. These studies predict a limited binary on/off encoding mechanism which underestimates the complexity of biological information processing, and hence the utility of single cell resolution data. Here we pursue a novel strategy that reformulates the information transfer problem as involving dynamic features of signaling rather than molecular abundances. We pursue a computational approach to test if and how the transcriptional regulatory activity patterns can be informative of the temporal history of signaling. Our analysis reveals (1) the dynamic features of signaling that significantly alter transcriptional regulatory patterns (encoding), and (2) the temporal history of signaling that can be inferred from single cell scale snapshots of transcriptional activity (decoding). Immediate early gene expression patterns were informative of signaling peak retention kinetics, whereas transcription factor activity patterns were informative of activation and deactivation kinetics of signaling. Moreover, the information processing aspects varied across the network, with each component encoding a selective subset of the dynamic signaling features. We developed novel sensitivity and information transfer maps to unravel the dynamic multiplexing of signaling features at each of these network components. Unsupervised clustering of the maps revealed two groups that aligned with network motifs distinguished by transcriptional feedforward vs feedback interactions. Our new computational methodology impacts the single cell scale experiments by identifying downstream snapshot measures required for inferring specific dynamical features of upstream signals involved in the regulation of cellular responses.

Project description:Natural vision engages a wide range of higher-level regions that integrate visual information over the large-scale brain network. How interareal connectivity reconfigures during the processing of ongoing natural visual scenes and how these dynamic functional changes relate to the underlaying anatomical links between regions is not well understood. Here, we hypothesized that macaque visual brain regions are poly-functional sharing the capacity to change their configuration state depending on the nature of visual input. To address this hypothesis, we reconstructed networks from in-vivo diffusion-weighted imaging (DWI) and functional magnetic resonance imaging (fMRI) data obtained in four alert macaque monkeys viewing naturalistic movie scenes. At first, we characterized network properties and found greater interhemispheric density and greater inter-subject variability in free-viewing networks as compared to structural networks. From the structural connectivity, we then captured modules on which we identified hubs during free-viewing that formed a widespread visuo-saccadic network across frontal (FEF, 46v), parietal (LIP, Tpt), and occipitotemporal modules (MT, V4, TEm), and that excluded primary visual cortex. Inter-subject variability of well-connected hubs reflected subject-specific configurations that largely recruited occipito-parietal and frontal modules. Across the cerebral hemispheres, free-viewing networks showed higher correlations among long-distance brain regions as compared to structural networks. From these findings, we hypothesized that long-distance interareal connectivity could reconfigure depending on the ongoing changes in visual scenes. Testing this hypothesis by applying temporally resolved functional connectivity we observed that many structurally defined areas (such as areas V4, MT/MST and LIP) were poly-functional as they were recruited as hub members of multiple network states that changed during the presentation of scenes containing objects, motion, faces, and actions. We suggest that functional flexibility in macaque macroscale brain networks is required for the efficient interareal communication during active natural vision. To further promote the use of naturalistic free-viewing paradigms and increase the development of macaque neuroimaging resources, we share our datasets in the PRIME-DE consortium.

Dataset Information

Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision.

Publications

Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets