Toward a universal decoder of linguistic meaning from brain activation.
Ontology highlight
ABSTRACT:
SUBMITTER: Pereira F
PROVIDER: S-EPMC5840373 | biostudies-literature | 2018 Mar
REPOSITORIES: biostudies-literature
Publications
Toward a universal decoder of linguistic meaning from brain activation.
Pereira Francisco F Lou Bin B Pritchett Brianna B Ritter Samuel S Gershman Samuel J SJ Kanwisher Nancy N Botvinick Matthew M Fedorenko Evelina E
Nature communications 20180306 1
Prior work decoding linguistic meaning from imaging data has been largely limited to concrete nouns, using similar stimuli for training and testing, from a relatively small number of semantic categories. Here we present a new approach for building a brain decoding system in which words and sentences are represented as vectors in a semantic space constructed from massive text corpora. By efficiently sampling this space to select training stimuli shown to subjects, we maximize the ability to gener ...[more]
Similar Datasets
Project description:The brain is thought to combine linguistic knowledge of words and nonlinguistic knowledge of their referents to encode sentence meaning. However, functional neuroimaging studies aiming at decoding language meaning from neural activity have mostly relied on distributional models of word semantics, which are based on patterns of word co-occurrence in text corpora. Here, we present initial evidence that modeling nonlinguistic "experiential" knowledge contributes to decoding neural representations of sentence meaning. We model attributes of peoples' sensory, motor, social, emotional, and cognitive experiences with words using behavioral ratings. We demonstrate that fMRI activation elicited in sentence reading is more accurately decoded when this experiential attribute model is integrated with a text-based model than when either model is applied in isolation (participants were 5 males and 9 females). Our decoding approach exploits a representation-similarity-based framework, which benefits from being parameter free, while performing at accuracy levels comparable with those from parameter fitting approaches, such as ridge regression. We find that the text-based model contributes particularly to the decoding of sentences containing linguistically oriented "abstract" words and reveal tentative evidence that the experiential model improves decoding of more concrete sentences. Finally, we introduce a cross-participant decoding method to estimate an upper bound on model-based decoding accuracy. We demonstrate that a substantial fraction of neural signal remains unexplained, and leverage this gap to pinpoint characteristics of weakly decoded sentences and hence identify model weaknesses to guide future model development.SIGNIFICANCE STATEMENT Language gives humans the unique ability to communicate about historical events, theoretical concepts, and fiction. Although words are learned through language and defined by their relations to other words in dictionaries, our understanding of word meaning presumably draws heavily on our nonlinguistic sensory, motor, interoceptive, and emotional experiences with words and their referents. Behavioral experiments lend support to the intuition that word meaning integrates aspects of linguistic and nonlinguistic "experiential" knowledge. However, behavioral measures do not provide a window on how meaning is represented in the brain and tend to necessitate artificial experimental paradigms. We present a model-based approach that reveals early evidence that experiential and linguistically acquired knowledge can be detected in brain activity elicited in reading natural sentences.
| S-EPMC6832686 | biostudies-literature
Project description:Recent research on human nonverbal vocalizations has led to considerable progress in our understanding of vocal communication of emotion. However, in contrast to studies of animal vocalizations, this research has focused mainly on the emotional interpretation of such signals. The repertoire of human nonverbal vocalizations as acoustic types, and the mapping between acoustic and emotional categories, thus remain underexplored. In a cross-linguistic naming task (Experiment 1), verbal categorization of 132 authentic (non-acted) human vocalizations by English-, Swedish- and Russian-speaking participants revealed the same major acoustic types: laugh, cry, scream, moan, and possibly roar and sigh. The association between call type and perceived emotion was systematic but non-redundant: listeners associated every call type with a limited, but in some cases relatively wide, range of emotions. The speed and consistency of naming the call type predicted the speed and consistency of inferring the caller's emotion, suggesting that acoustic and emotional categorizations are closely related. However, participants preferred to name the call type before naming the emotion. Furthermore, nonverbal categorization of the same stimuli in a triad classification task (Experiment 2) was more compatible with classification by call type than by emotion, indicating the former's greater perceptual salience. These results suggest that acoustic categorization may precede attribution of emotion, highlighting the need to distinguish between the overt form of nonverbal signals and their interpretation by the perceiver. Both within- and between-call acoustic variation can then be modeled explicitly, bringing research on human nonverbal vocalizations more in line with the work on animal communication.
| S-EPMC5816134 | biostudies-literature
Project description:Neuroprosthetic brain-computer interfaces are systems that decode neural activity into useful control signals for effectors, such as a cursor on a computer screen. It has long been recognized that both the user and decoding system can adapt to increase the accuracy of the end effector. Co-adaptation is the process whereby a user learns to control the system in conjunction with the decoder adapting to learn the user's neural patterns. We provide a mathematical framework for co-adaptation and relate co-adaptation to the joint optimization of the user's control scheme ("encoding model") and the decoding algorithm's parameters. When the assumptions of that framework are respected, co-adaptation cannot yield better performance than that obtainable by an optimal initial choice of fixed decoder, coupled with optimal user learning. For a specific case, we provide numerical methods to obtain such an optimized decoder. We demonstrate our approach in a model brain-computer interface system using an online prosthesis simulator, a simple human-in-the-loop pyschophysics setup which provides a non-invasive simulation of the BCI setting. These experiments support two claims: that users can learn encoders matched to fixed, optimal decoders and that, once learned, our approach yields expected performance advantages.
| S-EPMC4451011 | biostudies-literature
Project description:BackgroundLinguists and psychologists have explained the remarkable similarities in the orderings of linguistic elements across languages by suggesting that our inborn ability for language makes available certain innately wired primitives. Different types of adjectives, adverbs, and other elements in the functional spine are considered to occupy fixed positions via innate hierarchies that determine orderings such as A>B>C, banning other permutations (*B>C>A). The goal of this research is to tap into the nature and rigidity of such hierarchies by comparing what happens when people process orderings that either comply with them or violate them.MethodN = 170 neurotypical, adult speakers completed a timed forced choice task that featured stimuli showing a combination of two adjectives and a Spelke-object (e.g., 'I bought a square black table'). Two types of responses were collected: (i) acceptability judgments on a 3-point Likert scale that featured the options 'correct', 'neither correct nor wrong', and 'wrong' and (ii) reaction times. The task featured three conditions: 1. size adjective > nationality adjective, 2. color adjective > shape adjective, 3. subjective comment adjective > material adjective. Each condition had two orders. In the congruent order, the adjective pair was ordered in agreement with what is traditionally accepted as dictated by the universal hierarchy. In the incongruent order, the ordering was reversed, thus the hierarchy was violated.ResultsIn the first experiment, the results of n = 140 monolinguals showed that across conditions, both congruent and incongruent orders were generally accepted as correct. For 2/3 conditions, the difference in acceptability ratings between congruent and incongruent orders did not reach statistical significance. Using time as a window to processing, reaction times showed that incongruent orders do not take longer to process than congruent ones, as should be the case if the former were treated as being licensed under some type of special condition (e.g., contrastive focus) that reverses the unmarked order and legitimizes the violation of the hierarchy. In the second experiment, the results of n = 30 bidialectals, tested in both language varieties, corroborated the findings of the first experiment.ConclusionsOur findings do not provide evidence for an innate hierarchy for adjective ordering that imposes one rigid, unmarked order. We discuss the importance of notions such as subjectivity and inherentness, and show that for some conditions, not only is there no evidence for a hard constraint that bans incongruent orders, but even simple preferences of congruent orders over incongruent ones are hard to discern. Capturing the bigger picture, given that both the hierarchies and their legit permutations have been described as innate, our results reduce the amount of primitives that are cast as innate, eventually offering a deflationist approach to human linguistic cognition.
| S-EPMC6679903 | biostudies-literature
Project description:Bidirectional brain-machine interfaces (BMIs) establish a two-way direct communication link between the brain and the external world. A decoder translates recorded neural activity into motor commands and an encoder delivers sensory information collected from the environment directly to the brain creating a closed-loop system. These two modules are typically integrated in bulky external devices. However, the clinical support of patients with severe motor and sensory deficits requires compact, low-power, and fully implantable systems that can decode neural signals to control external devices. As a first step toward this goal, we developed a modular bidirectional BMI setup that uses a compact neuromorphic processor as a decoder. On this chip we implemented a network of spiking neurons built using its ultra-low-power mixed-signal analog/digital circuits. On-chip on-line spike-timing-dependent plasticity synapse circuits enabled the network to learn to decode neural signals recorded from the brain into motor outputs controlling the movements of an external device. The modularity of the BMI allowed us to tune the individual components of the setup without modifying the whole system. In this paper, we present the features of this modular BMI and describe how we configured the network of spiking neuron circuits to implement the decoder and to coordinate it with the encoder in an experimental BMI paradigm that connects bidirectionally the brain of an anesthetized rat with an external object. We show that the chip learned the decoding task correctly, allowing the interfaced brain to control the object's trajectories robustly. Based on our demonstration, we propose that neuromorphic technology is mature enough for the development of BMI modules that are sufficiently low-power and compact, while being highly computationally powerful and adaptive.
| S-EPMC5145890 | biostudies-literature
Project description:The dual threats posed by the COVID-19 pandemic and hospital-acquired infections (HAIs) have emphasized the urgent need for self-disinfecting materials for infection control. Despite their highly potent antimicrobial activity, the adoption of photoactive materials to reduce infection transmission in hospitals and related healthcare facilities has been severely hampered by the lack of scalable and cost-effective manufacturing, in which case high-volume production methods for fabricating aPDI-based materials are needed. To address this issue here, we examined the antimicrobial efficacy of a simple bicomponent spray coating composed of the commercially-available UV-photocrosslinkable polymer N-methyl-4(4'-formyl-styryl)pyridinium methosulfate acetal poly(vinyl alcohol) (SbQ-PVA) and one of three aPDI photosensitizers (PSs): zinc-tetra(4-N-methylpyridyl)porphine (ZnTMPyP4+), methylene blue (MB), and Rose Bengal (RB). We applied these photodynamic coatings, collectively termed SbQ-PVA/PS, to a variety of commercially available materials. Scanning electron microscopy (SEM) and time-of-flight secondary ion mass spectrometry (ToF-SIMS) confirmed the successful application of the coatings, while inductively coupled plasma-optical emission spectroscopy (ICP-OES) revealed a photosensitizer loading of 0.09-0.78 nmol PS/mg material. The antimicrobial efficacy of the coated materials was evaluated against methicillin-susceptible Staphylococcus aureus ATCC-29213 and human coronavirus strain HCoV-229E. Upon illumination with visible light (60 min, 400-700 nm, 65 ± 5 mW/cm2), the coated materials inactivated S. aureus by 97-99.999% and HCoV-229E by 92-99.999%, depending on the material and PS employed. Photobleaching studies employing HCoV-229E demonstrated detection limit inactivation (99.999%) even after exposure for 4 weeks to indoor ambient room lighting. Taken together, these results demonstrate the potential for photodynamic SbQ-PVA/PS coatings to be universally applied to a wide range of materials for effectively reducing pathogen transmission.
| S-EPMC8355428 | biostudies-literature
Project description:Brain-machine interfaces (BMIs) seek to connect brains with machines or computers directly, for application in areas such as prosthesis control. For this application, the accuracy of the decoding of movement intentions is crucial. We aim to improve accuracy by designing a better encoding model of primary motor cortical activity during hand movements and combining this with decoder engineering refinements, resulting in a new unscented Kalman filter based decoder, UKF2, which improves upon our previous unscented Kalman filter decoder, UKF1. The new encoding model includes novel acceleration magnitude, position-velocity interaction, and target-cursor-distance features (the decoder does not require target position as input, it is decoded). We add a novel probabilistic velocity threshold to better determine the user's intent to move. We combine these improvements with several other refinements suggested by others in the field. Data from two Rhesus monkeys indicate that the UKF2 generates offline reconstructions of hand movements (mean CC 0.851) significantly more accurately than the UKF1 (0.833) and the popular position-velocity Kalman filter (0.812). The encoding model of the UKF2 could predict the instantaneous firing rate of neurons (mean CC 0.210), given kinematic variables and past spiking, better than the encoding models of these two decoders (UKF1: 0.138, p-v Kalman: 0.098). In closed-loop experiments where each monkey controlled a computer cursor with each decoder in turn, the UKF2 facilitated faster task completion (mean 1.56 s vs. 2.05 s) and higher Fitts's Law bit rate (mean 0.738 bit/s vs. 0.584 bit/s) than the UKF1. These results suggest that the modeling and decoder engineering refinements of the UKF2 improve decoding performance. We believe they can be used to enhance other decoders as well.
| S-EPMC5177654 | biostudies-literature
Project description:A word like Huh?--used as a repair initiator when, for example, one has not clearly heard what someone just said--is found in roughly the same form and function in spoken languages across the globe. We investigate it in naturally occurring conversations in ten languages and present evidence and arguments for two distinct claims: that Huh? is universal, and that it is a word. In support of the first, we show that the similarities in form and function of this interjection across languages are much greater than expected by chance. In support of the second claim we show that it is a lexical, conventionalised form that has to be learnt, unlike grunts or emotional cries. We discuss possible reasons for the cross-linguistic similarity and propose an account in terms of convergent evolution. Huh? is a universal word not because it is innate but because it is shaped by selective pressures in an interactional environment that all languages share: that of other-initiated repair. Our proposal enhances evolutionary models of language change by suggesting that conversational infrastructure can drive the convergent cultural evolution of linguistic items.
| S-EPMC3832628 | biostudies-literature
Project description:By polling individual responses to hypothetical scenarios, valuation studies estimate population preferences toward health on a quality-adjusted life year (QALY) scale. The scenarios typically involve trade-offs in time (time trade-off (TTO)), risk (standard gamble (SG)), or number of persons affected (person trade-off (PTO)). This paper revisits the QALY assumptions and provides a coherent health econometric approach that unites TTO, SG, and PTO techniques under a common estimator. The proposed approach avoids the use of ratio statistics in QALY estimation and the common convention of arbitrarily changing trade-off responses. As an example, 34% of the TTO responses from the seminal Measurement and Valuation of Health study were changed in the original UK analysis, which led to substantially lower QALY estimates. As a general rule, if the original estimate is less than 0.5 QALYs, add 0.25 QALYs to get the new estimates.
| S-EPMC3819815 | biostudies-literature
Project description:OBJECTIVE:Recent reports indicate that making better assumptions about the user's intended movement can improve the accuracy of decoder calibration for intracortical brain-computer interfaces. Several methods now exist for estimating user intent, including an optimal feedback control model, a piecewise-linear feedback control model, ReFIT, and other heuristics. Which of these methods yields the best decoding performance? METHODS:Using data from the BrainGate2 pilot clinical trial, we measured how a steady-state velocity Kalman filter decoder was affected by the choice of intention estimation method. We examined three separate components of the Kalman filter: dimensionality reduction, temporal smoothing, and output gain (speed scaling). RESULTS:The decoder's dimensionality reduction properties were largely unaffected by the intention estimation method. Decoded velocity vectors differed by <5% in terms of angular error and speed vs. target distance curves across methods. In contrast, the smoothing and gain properties of the decoder were greatly affected (> 50% difference in average values). Since the optimal gain and smoothing properties are task-specific (e.g. lower gains are better for smaller targets but worse for larger targets), no one method was better for all tasks. CONCLUSION:Our results show that, when gain and smoothing differences are accounted for, current intention estimation methods yield nearly equivalent decoders and that simple models of user intent, such as a position error vector (target position minus cursor position), perform comparably to more elaborate models. Our results also highlight that simple differences in gain and smoothing properties have a large effect on online performance and can confound decoder comparisons.
| S-EPMC6043406 | biostudies-literature