Toward a universal decoder of linguistic meaning from brain activation.
Ontology highlight
ABSTRACT:
SUBMITTER: Pereira F
PROVIDER: S-EPMC5840373 | biostudies-literature | 2018 Mar
REPOSITORIES: biostudies-literature
Publications
Toward a universal decoder of linguistic meaning from brain activation.
Pereira Francisco F Lou Bin B Pritchett Brianna B Ritter Samuel S Gershman Samuel J SJ Kanwisher Nancy N Botvinick Matthew M Fedorenko Evelina E
Nature communications 20180306 1
Prior work decoding linguistic meaning from imaging data has been largely limited to concrete nouns, using similar stimuli for training and testing, from a relatively small number of semantic categories. Here we present a new approach for building a brain decoding system in which words and sentences are represented as vectors in a semantic space constructed from massive text corpora. By efficiently sampling this space to select training stimuli shown to subjects, we maximize the ability to gener ...[more]
Similar Datasets
Project description:The brain is thought to combine linguistic knowledge of words and nonlinguistic knowledge of their referents to encode sentence meaning. However, functional neuroimaging studies aiming at decoding language meaning from neural activity have mostly relied on distributional models of word semantics, which are based on patterns of word co-occurrence in text corpora. Here, we present initial evidence that modeling nonlinguistic "experiential" knowledge contributes to decoding neural representations of sentence meaning. We model attributes of peoples' sensory, motor, social, emotional, and cognitive experiences with words using behavioral ratings. We demonstrate that fMRI activation elicited in sentence reading is more accurately decoded when this experiential attribute model is integrated with a text-based model than when either model is applied in isolation (participants were 5 males and 9 females). Our decoding approach exploits a representation-similarity-based framework, which benefits from being parameter free, while performing at accuracy levels comparable with those from parameter fitting approaches, such as ridge regression. We find that the text-based model contributes particularly to the decoding of sentences containing linguistically oriented "abstract" words and reveal tentative evidence that the experiential model improves decoding of more concrete sentences. Finally, we introduce a cross-participant decoding method to estimate an upper bound on model-based decoding accuracy. We demonstrate that a substantial fraction of neural signal remains unexplained, and leverage this gap to pinpoint characteristics of weakly decoded sentences and hence identify model weaknesses to guide future model development.SIGNIFICANCE STATEMENT Language gives humans the unique ability to communicate about historical events, theoretical concepts, and fiction. Although words are learned through language and defined by their relations to other words in dictionaries, our understanding of word meaning presumably draws heavily on our nonlinguistic sensory, motor, interoceptive, and emotional experiences with words and their referents. Behavioral experiments lend support to the intuition that word meaning integrates aspects of linguistic and nonlinguistic "experiential" knowledge. However, behavioral measures do not provide a window on how meaning is represented in the brain and tend to necessitate artificial experimental paradigms. We present a model-based approach that reveals early evidence that experiential and linguistically acquired knowledge can be detected in brain activity elicited in reading natural sentences.
| S-EPMC6832686 | biostudies-literature
Project description:Recent research on human nonverbal vocalizations has led to considerable progress in our understanding of vocal communication of emotion. However, in contrast to studies of animal vocalizations, this research has focused mainly on the emotional interpretation of such signals. The repertoire of human nonverbal vocalizations as acoustic types, and the mapping between acoustic and emotional categories, thus remain underexplored. In a cross-linguistic naming task (Experiment 1), verbal categorization of 132 authentic (non-acted) human vocalizations by English-, Swedish- and Russian-speaking participants revealed the same major acoustic types: laugh, cry, scream, moan, and possibly roar and sigh. The association between call type and perceived emotion was systematic but non-redundant: listeners associated every call type with a limited, but in some cases relatively wide, range of emotions. The speed and consistency of naming the call type predicted the speed and consistency of inferring the caller's emotion, suggesting that acoustic and emotional categorizations are closely related. However, participants preferred to name the call type before naming the emotion. Furthermore, nonverbal categorization of the same stimuli in a triad classification task (Experiment 2) was more compatible with classification by call type than by emotion, indicating the former's greater perceptual salience. These results suggest that acoustic categorization may precede attribution of emotion, highlighting the need to distinguish between the overt form of nonverbal signals and their interpretation by the perceiver. Both within- and between-call acoustic variation can then be modeled explicitly, bringing research on human nonverbal vocalizations more in line with the work on animal communication.
| S-EPMC5816134 | biostudies-literature
Project description:Neuroprosthetic brain-computer interfaces are systems that decode neural activity into useful control signals for effectors, such as a cursor on a computer screen. It has long been recognized that both the user and decoding system can adapt to increase the accuracy of the end effector. Co-adaptation is the process whereby a user learns to control the system in conjunction with the decoder adapting to learn the user's neural patterns. We provide a mathematical framework for co-adaptation and relate co-adaptation to the joint optimization of the user's control scheme ("encoding model") and the decoding algorithm's parameters. When the assumptions of that framework are respected, co-adaptation cannot yield better performance than that obtainable by an optimal initial choice of fixed decoder, coupled with optimal user learning. For a specific case, we provide numerical methods to obtain such an optimized decoder. We demonstrate our approach in a model brain-computer interface system using an online prosthesis simulator, a simple human-in-the-loop pyschophysics setup which provides a non-invasive simulation of the BCI setting. These experiments support two claims: that users can learn encoders matched to fixed, optimal decoders and that, once learned, our approach yields expected performance advantages.
| S-EPMC4451011 | biostudies-literature
Project description:Recent research has established that cortical activity "tracks" the presentation rate of syntactic phrases in continuous speech, even though phrases are abstract units that do not have direct correlates in the acoustic signal. We investigated whether cortical tracking of phrase structures is modulated by the extent to which these structures compositionally determine meaning. To this end, we recorded electroencephalography (EEG) of 38 native speakers who listened to naturally spoken Dutch stimuli in different conditions, which parametrically modulated the degree to which syntactic structure and lexical semantics determine sentence meaning. Tracking was quantified through mutual information between the EEG data and either the speech envelopes or abstract annotations of syntax, all of which were filtered in the frequency band corresponding to the presentation rate of phrases (1.1-2.1 Hz). Overall, these mutual information analyses showed stronger tracking of phrases in regular sentences than in stimuli whose lexical-syntactic content is reduced, but no consistent differences in tracking between sentences and stimuli that contain a combination of syntactic structure and lexical content. While there were no effects of compositional meaning on the degree of phrase-structure tracking, analyses of event-related potentials elicited by sentence-final words did reveal meaning-induced differences between conditions. Our findings suggest that cortical tracking of structure in sentences indexes the internal generation of this structure, a process that is modulated by the properties of its input, but not by the compositional interpretation of its output.
| S-EPMC10158633 | biostudies-literature
Project description:The visual system provides a valuable model for studying the working mechanisms of sensory processing and high-level consciousness. A significant challenge in this field is the reconstruction of images from decoded neural activity, which could not only test the accuracy of our understanding of the visual system but also provide a practical tool for solving real-world problems. Although recent advances in deep learning have improved the decoding of neural spike trains, little attention has been paid to the underlying mechanisms of the visual system. To address this issue, we propose a deep learning neural network architecture that incorporates the biological properties of the visual system, such as receptive fields, to reconstruct visual images from spike trains. Our model outperforms current models and has been evaluated on different datasets from both retinal ganglion cells (RGCs) and the primary visual cortex (V1) neural spikes. Our model demonstrated the great potential of brain-inspired algorithms to solve a challenge that our brain solves.
| S-EPMC10185745 | biostudies-literature
Project description:BackgroundLinguists and psychologists have explained the remarkable similarities in the orderings of linguistic elements across languages by suggesting that our inborn ability for language makes available certain innately wired primitives. Different types of adjectives, adverbs, and other elements in the functional spine are considered to occupy fixed positions via innate hierarchies that determine orderings such as A>B>C, banning other permutations (*B>C>A). The goal of this research is to tap into the nature and rigidity of such hierarchies by comparing what happens when people process orderings that either comply with them or violate them.MethodN = 170 neurotypical, adult speakers completed a timed forced choice task that featured stimuli showing a combination of two adjectives and a Spelke-object (e.g., 'I bought a square black table'). Two types of responses were collected: (i) acceptability judgments on a 3-point Likert scale that featured the options 'correct', 'neither correct nor wrong', and 'wrong' and (ii) reaction times. The task featured three conditions: 1. size adjective > nationality adjective, 2. color adjective > shape adjective, 3. subjective comment adjective > material adjective. Each condition had two orders. In the congruent order, the adjective pair was ordered in agreement with what is traditionally accepted as dictated by the universal hierarchy. In the incongruent order, the ordering was reversed, thus the hierarchy was violated.ResultsIn the first experiment, the results of n = 140 monolinguals showed that across conditions, both congruent and incongruent orders were generally accepted as correct. For 2/3 conditions, the difference in acceptability ratings between congruent and incongruent orders did not reach statistical significance. Using time as a window to processing, reaction times showed that incongruent orders do not take longer to process than congruent ones, as should be the case if the former were treated as being licensed under some type of special condition (e.g., contrastive focus) that reverses the unmarked order and legitimizes the violation of the hierarchy. In the second experiment, the results of n = 30 bidialectals, tested in both language varieties, corroborated the findings of the first experiment.ConclusionsOur findings do not provide evidence for an innate hierarchy for adjective ordering that imposes one rigid, unmarked order. We discuss the importance of notions such as subjectivity and inherentness, and show that for some conditions, not only is there no evidence for a hard constraint that bans incongruent orders, but even simple preferences of congruent orders over incongruent ones are hard to discern. Capturing the bigger picture, given that both the hierarchies and their legit permutations have been described as innate, our results reduce the amount of primitives that are cast as innate, eventually offering a deflationist approach to human linguistic cognition.
| S-EPMC6679903 | biostudies-literature
Project description:Bidirectional brain-machine interfaces (BMIs) establish a two-way direct communication link between the brain and the external world. A decoder translates recorded neural activity into motor commands and an encoder delivers sensory information collected from the environment directly to the brain creating a closed-loop system. These two modules are typically integrated in bulky external devices. However, the clinical support of patients with severe motor and sensory deficits requires compact, low-power, and fully implantable systems that can decode neural signals to control external devices. As a first step toward this goal, we developed a modular bidirectional BMI setup that uses a compact neuromorphic processor as a decoder. On this chip we implemented a network of spiking neurons built using its ultra-low-power mixed-signal analog/digital circuits. On-chip on-line spike-timing-dependent plasticity synapse circuits enabled the network to learn to decode neural signals recorded from the brain into motor outputs controlling the movements of an external device. The modularity of the BMI allowed us to tune the individual components of the setup without modifying the whole system. In this paper, we present the features of this modular BMI and describe how we configured the network of spiking neuron circuits to implement the decoder and to coordinate it with the encoder in an experimental BMI paradigm that connects bidirectionally the brain of an anesthetized rat with an external object. We show that the chip learned the decoding task correctly, allowing the interfaced brain to control the object's trajectories robustly. Based on our demonstration, we propose that neuromorphic technology is mature enough for the development of BMI modules that are sufficiently low-power and compact, while being highly computationally powerful and adaptive.
| S-EPMC5145890 | biostudies-literature
Project description: Not available
| S-EPMC6440568 | biostudies-literature
Project description:The dual threats posed by the COVID-19 pandemic and hospital-acquired infections (HAIs) have emphasized the urgent need for self-disinfecting materials for infection control. Despite their highly potent antimicrobial activity, the adoption of photoactive materials to reduce infection transmission in hospitals and related healthcare facilities has been severely hampered by the lack of scalable and cost-effective manufacturing, in which case high-volume production methods for fabricating aPDI-based materials are needed. To address this issue here, we examined the antimicrobial efficacy of a simple bicomponent spray coating composed of the commercially-available UV-photocrosslinkable polymer N-methyl-4(4'-formyl-styryl)pyridinium methosulfate acetal poly(vinyl alcohol) (SbQ-PVA) and one of three aPDI photosensitizers (PSs): zinc-tetra(4-N-methylpyridyl)porphine (ZnTMPyP4+), methylene blue (MB), and Rose Bengal (RB). We applied these photodynamic coatings, collectively termed SbQ-PVA/PS, to a variety of commercially available materials. Scanning electron microscopy (SEM) and time-of-flight secondary ion mass spectrometry (ToF-SIMS) confirmed the successful application of the coatings, while inductively coupled plasma-optical emission spectroscopy (ICP-OES) revealed a photosensitizer loading of 0.09-0.78 nmol PS/mg material. The antimicrobial efficacy of the coated materials was evaluated against methicillin-susceptible Staphylococcus aureus ATCC-29213 and human coronavirus strain HCoV-229E. Upon illumination with visible light (60 min, 400-700 nm, 65 ± 5 mW/cm2), the coated materials inactivated S. aureus by 97-99.999% and HCoV-229E by 92-99.999%, depending on the material and PS employed. Photobleaching studies employing HCoV-229E demonstrated detection limit inactivation (99.999%) even after exposure for 4 weeks to indoor ambient room lighting. Taken together, these results demonstrate the potential for photodynamic SbQ-PVA/PS coatings to be universally applied to a wide range of materials for effectively reducing pathogen transmission.
| S-EPMC8355428 | biostudies-literature
Project description:Brain-machine interfaces (BMIs) seek to connect brains with machines or computers directly, for application in areas such as prosthesis control. For this application, the accuracy of the decoding of movement intentions is crucial. We aim to improve accuracy by designing a better encoding model of primary motor cortical activity during hand movements and combining this with decoder engineering refinements, resulting in a new unscented Kalman filter based decoder, UKF2, which improves upon our previous unscented Kalman filter decoder, UKF1. The new encoding model includes novel acceleration magnitude, position-velocity interaction, and target-cursor-distance features (the decoder does not require target position as input, it is decoded). We add a novel probabilistic velocity threshold to better determine the user's intent to move. We combine these improvements with several other refinements suggested by others in the field. Data from two Rhesus monkeys indicate that the UKF2 generates offline reconstructions of hand movements (mean CC 0.851) significantly more accurately than the UKF1 (0.833) and the popular position-velocity Kalman filter (0.812). The encoding model of the UKF2 could predict the instantaneous firing rate of neurons (mean CC 0.210), given kinematic variables and past spiking, better than the encoding models of these two decoders (UKF1: 0.138, p-v Kalman: 0.098). In closed-loop experiments where each monkey controlled a computer cursor with each decoder in turn, the UKF2 facilitated faster task completion (mean 1.56 s vs. 2.05 s) and higher Fitts's Law bit rate (mean 0.738 bit/s vs. 0.584 bit/s) than the UKF1. These results suggest that the modeling and decoder engineering refinements of the UKF2 improve decoding performance. We believe they can be used to enhance other decoders as well.
| S-EPMC5177654 | biostudies-literature