Dataset Information

Unsupervised Decoding of Long-Term, Naturalistic Human Neural Recordings with Automated Video and Audio Annotations.

ABSTRACT: Fully automated decoding of human activities and intentions from direct neural recordings is a tantalizing challenge in brain-computer interfacing. Implementing Brain Computer Interfaces (BCIs) outside carefully controlled experiments in laboratory settings requires adaptive and scalable strategies with minimal supervision. Here we describe an unsupervised approach to decoding neural states from naturalistic human brain recordings. We analyzed continuous, long-term electrocorticography (ECoG) data recorded over many days from the brain of subjects in a hospital room, with simultaneous audio and video recordings. We discovered coherent clusters in high-dimensional ECoG recordings using hierarchical clustering and automatically annotated them using speech and movement labels extracted from audio and video. To our knowledge, this represents the first time techniques from computer vision and speech processing have been used for natural ECoG decoding. Interpretable behaviors were decoded from ECoG data, including moving, speaking and resting; the results were assessed by comparison with manual annotation. Discovered clusters were projected back onto the brain revealing features consistent with known functional areas, opening the door to automated functional brain mapping in natural settings.

SUBMITTER: Wang NX

PROVIDER: S-EPMC4838634 | biostudies-literature | 2016

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Unsupervised Decoding of Long-Term, Naturalistic Human Neural Recordings with Automated Video and Audio Annotations.

Wang Nancy X R NX Olson Jared D JD Ojemann Jeffrey G JG Rao Rajesh P N RP Brunton Bingni W BW

Frontiers in human neuroscience 20160421

Fully automated decoding of human activities and intentions from direct neural recordings is a tantalizing challenge in brain-computer interfacing. Implementing Brain Computer Interfaces (BCIs) outside carefully controlled experiments in laboratory settings requires adaptive and scalable strategies with minimal supervision. Here we describe an unsupervised approach to decoding neural states from naturalistic human brain recordings. We analyzed continuous, long-term electrocorticography (ECoG) da ...[more]

PMID: 27148018

Similar Datasets

Project description:Material properties, such as softness or stickiness, determine how an object can be used. Based on our real-life experience, we form strong expectations about how objects should behave under force, given their typical material properties. Such expectations have been shown to modulate perceptual processes, but we currently do not know how expectation influences the temporal dynamics of the cortical visual analysis for objects and their materials. Here, we tracked the neural representations of expected and unexpected material behaviors using time-resolved EEG decoding in a violation-of-expectation paradigm, where objects fell to the ground and deformed in expected or unexpected ways. Participants were 25 men and women. Our study yielded three key results: First, both objects and materials were represented rapidly and in a temporally sustained fashion. Second, objects exhibiting unexpected material behaviors were more successfully decoded than objects exhibiting expected behaviors within 190 ms after the impact, which might indicate additional processing demands when expectations are unmet. Third, general signals of expectation fulfillment that generalize across specific objects and materials were found within the first 150 ms after the impact. Together, our results provide new insights into the temporal neural processing cascade that underlies the analysis of real-world material behaviors. They reveal a sequence of predictions, with cortical signals progressing from a general signature of expectation fulfillment toward increased processing of unexpected material behaviors.SIGNIFICANCE STATEMENT In the real world, we can make accurate predictions about how an object's material shapes its behavior: For instance, we know that cups are typically made of porcelain and shatter when we accidentally drop them. Here, we use EEG to experimentally test how expectations about material behaviors impact neural processing. We showed our participants videos of objects that exhibited expected material behaviors (e.g., a glass shattering when falling to the ground) or unexpected material behaviors (e.g., a glass melting on impact). Our results reveal a hierarchy of predictions in cortex: The visual system rapidly generates signals that index whether expectations about material behaviors are met. These signals are followed by increased processing of objects displaying unexpected material behaviors.

Project description:Objective. Electrical recordings of neural activity from brain surface have been widely employed in basic neuroscience research and clinical practice for investigations of neural circuit functions, brain-computer interfaces, and treatments for neurological disorders. Traditionally, these surface potentials have been believed to mainly reflect local neural activity. It is not known how informative the locally recorded surface potentials are for the neural activities across multiple cortical regions.Approach. To investigate that, we perform simultaneous local electrical recording and wide-field calcium imaging in awake head-fixed mice. Using a recurrent neural network model, we try to decode the calcium fluorescence activity of multiple cortical regions from local electrical recordings.Main results. The mean activity of different cortical regions could be decoded from locally recorded surface potentials. Also, each frequency band of surface potentials differentially encodes activities from multiple cortical regions so that including all the frequency bands in the decoding model gives the highest decoding performance. Despite the close spacing between recording channels, surface potentials from different channels provide complementary information about the large-scale cortical activity and the decoding performance continues to improve as more channels are included. Finally, we demonstrate the successful decoding of whole dorsal cortex activity at pixel-level using locally recorded surface potentials.Significance. These results show that the locally recorded surface potentials indeed contain rich information of the large-scale neural activities, which could be further demixed to recover the neural activity across individual cortical regions. In the future, our cross-modality inference approach could be adapted to virtually reconstruct cortex-wide brain activity, greatly expanding the spatial reach of surface electrical recordings without increasing invasiveness. Furthermore, it could be used to facilitate imaging neural activity across the whole cortex in freely moving animals, without requirement of head-fixed microscopy configurations.

Dataset Information

Unsupervised Decoding of Long-Term, Naturalistic Human Neural Recordings with Automated Video and Audio Annotations.

Publications

Unsupervised Decoding of Long-Term, Naturalistic Human Neural Recordings with Automated Video and Audio Annotations.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets