Browse
Submit Data
Databases
API
Help

Dataset Information

0 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Cross-modal noise compensation in audiovisual words.

ABSTRACT: Perceiving linguistic input is vital for human functioning, but the process is complicated by the fact that the incoming signal is often degraded. However, humans can compensate for unimodal noise by relying on simultaneous sensory input from another modality. Here, we investigated noise-compensation for spoken and printed words in two experiments. In the first behavioral experiment, we observed that accuracy was modulated by reaction time, bias and sensitivity, but noise compensation could nevertheless be explained via accuracy differences when controlling for RT, bias and sensitivity. In the second experiment, we also measured Event Related Potentials (ERPs) and observed robust electrophysiological correlates of noise compensation starting at around 350?ms after stimulus onset, indicating that noise compensation is most prominent at lexical/semantic processing levels.

SUBMITTER: Baart M

PROVIDER: S-EPMC5294401 | biostudies-literature | 2017 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Publications

Cross-modal noise compensation in audiovisual words.

Baart Martijn M Armstrong Blair C BC Martin Clara D CD Frost Ram R Carreiras Manuel M

Scientific reports 20170207

Perceiving linguistic input is vital for human functioning, but the process is complicated by the fact that the incoming signal is often degraded. However, humans can compensate for unimodal noise by relying on simultaneous sensory input from another modality. Here, we investigated noise-compensation for spoken and printed words in two experiments. In the first behavioral experiment, we observed that accuracy was modulated by reaction time, bias and sensitivity, but noise compensation could neve ...[more]

PMID: 28169316

Similar Datasets

The neural representations underlying asymmetric cross-modal prediction of words.

Project description:Cross-modal prediction serves a crucial adaptive role in the multisensory world, yet the neural mechanisms underlying this prediction are poorly understood. The present study addressed this important question by combining a novel audiovisual sequence memory task, functional magnetic resonance imaging (fMRI), and multivariate neural representational analyses. Our behavioral results revealed a reliable asymmetric cross-modal predictive effect, with a stronger prediction from visual to auditory (VA) modality than auditory to visual (AV) modality. Mirroring the behavioral pattern, we found the superior parietal lobe (SPL) showed higher pattern similarity for VA than AV pairs, and the strength of the predictive coding in the SPL was positively correlated with the behavioral predictive effect in the VA condition. Representational connectivity analyses further revealed that the SPL mediated the neural pathway from the visual to the auditory cortex in the VA condition but was not involved in the auditory to visual cortex pathway in the AV condition. Direct neural pathways within the unimodal regions were found for the visual-to-visual and auditory-to-auditory predictions. Together, these results provide novel insights into the neural mechanisms underlying cross-modal sequence prediction.

| S-EPMC10028649 | biostudies-literature

Audiovisual Emotional Congruency Modulates the Stimulus-Driven Cross-Modal Spread of Attention.

Project description:It has been reported that attending to stimuli in visual modality can spread to task-irrelevant but synchronously presented stimuli in auditory modality, a phenomenon termed the cross-modal spread of attention, which could be either stimulus-driven or representation-driven depending on whether the visual constituent of an audiovisual object is further selected based on the object representation. The stimulus-driven spread of attention occurs whenever a task-irrelevant sound synchronizes with an attended visual stimulus, regardless of the cross-modal semantic congruency. The present study recorded event-related potentials (ERPs) to investigate whether the stimulus-driven cross-modal spread of attention could be modulated by audio-visual emotional congruency in a visual oddball task where emotion (positive/negative) was task-irrelevant. The results first demonstrated a prominent stimulus-driven spread of attention regardless of audio-visual emotional congruency by showing that for all audiovisual pairs, the extracted ERPs to the auditory constituents of audiovisual stimuli within the time window of 200-300 ms were significantly larger than ERPs to the same auditory stimuli delivered alone. However, the amplitude of this stimulus-driven auditory Nd component during 200-300 ms was significantly larger for emotionally incongruent than congruent audiovisual stimuli when their visual constituents' emotional valences were negative. Moreover, the Nd was sustained during 300-400 ms only for the incongruent audiovisual stimuli with emotionally negative visual constituents. These findings suggest that although the occurrence of the stimulus-driven cross-modal spread of attention is independent of audio-visual emotional congruency, its magnitude is nevertheless modulated even when emotion is task-irrelevant.

| S-EPMC9497153 | biostudies-literature

Cross-modal metaphorical mapping of spoken emotion words onto vertical space.

Project description:From the field of embodied cognition, previous studies have reported evidence of metaphorical mapping of emotion concepts onto a vertical spatial axis. Most of the work on this topic has used visual words as the typical experimental stimuli. However, to our knowledge, no previous study has examined the association between affect and vertical space using a cross-modal procedure. The current research is a first step toward the study of the metaphorical mapping of emotions onto vertical space by means of an auditory to visual cross-modal paradigm. In the present study, we examined whether auditory words with an emotional valence can interact with the vertical visual space according to a 'positive-up/negative-down' embodied metaphor. The general method consisted in the presentation of a spoken word denoting a positive/negative emotion prior to the spatial localization of a visual target in an upper or lower position. In Experiment 1, the spoken words were passively heard by the participants and no reliable interaction between emotion concepts and bodily simulated space was found. In contrast, Experiment 2 required more active listening of the auditory stimuli. A metaphorical mapping of affect and space was evident but limited to the participants engaged in an emotion-focused task. Our results suggest that the association of affective valence and vertical space is not activated automatically during speech processing since an explicit semantic and/or emotional evaluation of the emotionally valenced stimuli was necessary to obtain an embodied effect. The results are discussed within the framework of the embodiment hypothesis.

| S-EPMC4531208 | biostudies-literature

Cross-modal correspondence between auditory pitch and visual elevation modulates audiovisual temporal recalibration.

Project description:Cross-modal correspondences refer to associations between feature dimensions of stimuli across sensory modalities. Research has indicated that correspondence between audiovisual stimuli influences whether these stimuli are integrated or segregated. On the other hand, the audiovisual integration process plastically changes to compensate for continuously observed spatiotemporal conflicts between sensory modalities. If and how cross-modal correspondence modulates the "recalibration" of integration is unclear. We investigated whether cross-modal correspondence between auditory pitch and visual elevation affected audiovisual temporal recalibration. Participants judged the simultaneity of a pair of audiovisual stimuli after an adaptation phase in which alternating auditory and visual stimuli equally spaced in time were presented. In the adaptation phase, auditory pitch and visual elevation were manipulated to fix the order within each pairing of audiovisual stimuli congruent with pitch-elevation correspondence (visual leading or auditory leading). We found a shift in the point of subjective simultaneity (PSS) between congruent audiovisual stimuli as a function of the adaptation conditions (Experiment 1, 2), but this shift in the PSS was not observed within incongruent pairs (Experiment 2). These results indicate that asynchronies between audiovisual signals congruent with cross-modal correspondence are selectively recalibrated.

| S-EPMC9734665 | biostudies-literature

Implicit Association Test (IAT) Studies Investigating Pitch-Shape Audiovisual Cross-modal Associations Across Language Groups.

Project description:Previous studies have shown that Chinese speakers and non-Chinese speakers exhibit different patterns of cross-modal congruence for the lexical tones of Mandarin Chinese, depending on which features of the pitch they attend to. But is this pattern of language-specific listening a conscious cultural strategy or an automatic processing effect? If automatic, does it also apply when the same pitch contours no longer sound like speech? Implicit Association Tests (IATs) provide an indirect measure of cross-modal association. In a series of IAT studies, conducted with participants with three kinds of language backgrounds (Chinese-dominant bilinguals, Chinese balanced bilinguals, and English speakers with no Chinese experience) we find language-specific congruence effects for Mandarin lexical tones but not for matched sine-wave stimuli. That is, for linguistic stimuli, non-Chinese speakers show advantages for pitch-height congruence (high-pointy, low-curvy); no congruence effects were found for Chinese speakers. For non-linguistic stimuli, all participant groups showed advantages for pitch-height congruence. The present findings suggest that non-lexical tone congruence (high-pointy, low-curvy) is a basic congruence pattern, and the acquisition of a language with lexical tone can alter this perception.

| S-EPMC10078355 | biostudies-literature

The impact of spoken action words on performance in a cross-modal oddball task.

Project description:In this study a cross-modal oddball task was employed to study the effect that words spoken either non-urgently or urgently would have on a digit categorization task and if women would exhibit greater behavioral inhibitory control. The words were unrelated to the task itself, but related to the action required to complete the task. Forty participants (21 women) conducted a computerized categorization task while exposed to a sinewave tone as a standard stimulus (75% of the trials) or a to-be ignored word (press, stop) spoken either non-urgently or urgently as unexpected auditory deviant stimulus (6.25% trials for each category). Urgent words had sharp intonation and an average fundamental frequency (F0) ranging from 191.9 (stop) to 204.6 (press) Hz. Non-urgent words had low intonation with average F0 ranging from 103.9.9 (stop) to 120.3 (press) Hz. As expected, deviant distraction and longer response times were found by exposure to the word stop, but deviant distraction was not found to be significant with the word press or due to intonation. While the results showed that women had in general longer reaction times, there were no gender differences found related to the deviant distraction caused by word or intonation. The present results do not support the hypothesis that women have greater behavioral inhibitory control, but there was evidence that the meaning of the word could influence response times.

| S-EPMC6245795 | biostudies-literature

Arabic digits and spoken number words: Timing modulates the cross-modal numerical distance effect.

Project description:Moving seamlessly between spoken number words and Arabic digits is common in everyday life. In this study, we systematically investigated the correspondence between auditory number words and visual Arabic digits in adults. Auditory number words and visual Arabic digits were presented concurrently or sequentially and participants had to indicate whether they described the same quantity. We manipulated the stimulus onset asynchronies (SOAs) between the two stimuli (Experiment 1: -500 ms to +500 ms; Experiment 2: -200 ms to +200 ms). In both experiments, we found a significant cross-modal distance effect. This effect was strongest for simultaneous stimulus presentation and decreased with increasing SOAs. Numerical distance emerged as the most consistent significant predictor overall, in particular for simultaneous presentation. However, physical similarity between the stimuli was often a significant predictor of response times in addition to numerical distance, and at longer SOAs, physical similarity between the stimuli was the only significant predictor. This shows that SOA modulates the extent to which participants access quantity representations. Our results thus support the idea that a semantic quantity representation of auditory and visual numerical symbols is activated when participants perform a concurrent matching task, while at longer SOAs participants are more likely to rely on physical similarity between the stimuli. We also investigated whether individual differences in the efficiency of the cross-modal processing were related to differences in mathematical performance. Our results are inconclusive about whether the efficiency of cross-format numerical correspondence is related to mathematical competence in adults.

| S-EPMC6779017 | biostudies-other

Motor engagement relates to accurate perception of phonemes and audiovisual words, but not auditory words.

Project description:A longstanding debate has surrounded the role of the motor system in speech perception, but progress in this area has been limited by tasks that only examine isolated syllables and conflate decision-making with perception. Using an adaptive task that temporally isolates perception from decision-making, we examined an EEG signature of motor activity (sensorimotor μ/beta suppression) during the perception of auditory phonemes, auditory words, audiovisual words, and environmental sounds while holding difficulty constant at two levels (Easy/Hard). Results revealed left-lateralized sensorimotor μ/beta suppression that was related to perception of speech but not environmental sounds. Audiovisual word and phoneme stimuli showed enhanced left sensorimotor μ/beta suppression for correct relative to incorrect trials, while auditory word stimuli showed enhanced suppression for incorrect trials. Our results demonstrate that motor involvement in perception is left-lateralized, is specific to speech stimuli, and it not simply the result of domain-general processes. These results provide evidence for an interactive network for speech perception in which dorsal stream motor areas are dynamically engaged during the perception of speech depending on the characteristics of the speech signal. Crucially, this motor engagement has different effects on the perceptual outcome depending on the lexicality and modality of the speech stimulus.

| S-EPMC7835217 | biostudies-literature

Cross-modal and cross-language activation in bilinguals reveals lexical competition even when words or signs are unheard or unseen.

Project description:We exploit the phenomenon of cross-modal, cross-language activation to examine the dynamics of language processing. Previous within-language work showed that seeing a sign coactivates phonologically related signs, just as hearing a spoken word coactivates phonologically related words. In this study, we conducted a series of eye-tracking experiments using the visual world paradigm to investigate the time course of cross-language coactivation in hearing bimodal bilinguals (Spanish-Spanish Sign Language) and unimodal bilinguals (Spanish/Basque). The aim was to gauge whether (and how) seeing a sign could coactivate words and, conversely, how hearing a word could coactivate signs and how such cross-language coactivation patterns differ from within-language coactivation. The results revealed cross-language, cross-modal activation in both directions. Furthermore, comparison with previous findings of within-language lexical coactivation for spoken and signed language showed how the impact of temporal structure changes in different modalities. Spoken word activation follows the temporal structure of that word only when the word itself is heard; for signs, the temporal structure of the sign does not govern the time course of lexical access (location coactivation precedes handshape coactivation)-even when the sign is seen. We provide evidence that, instead, this pattern of activation is motivated by how common in the lexicon the sublexical units of the signs are. These results reveal the interaction between the perceptual properties of the explicit signal and structural linguistic properties. Examining languages across modalities illustrates how this interaction impacts language processing.

| S-EPMC9457174 | biostudies-literature

Finding a home in the noise: cross-modal impact of anthropogenic vibration on animal search behaviour.

Project description:Chemical cues and signals enable animals to sense their surroundings over vast distances and find key resources, like food and shelter. However, the use of chemosensory information may be impaired in aquatic habitats by anthropogenic activities, which produce both water-borne sounds and substrate-borne vibrations, potentially affecting not only vibroacoustic sensing but other modalities as well. We attracted marine hermit crabs (Pagurus acadianus) in field experiments using a chemical cue indicative of a newly available shell home. We then quantified the number of crabs arriving in control versus impulsive noise conditions. Treatment (control or noise), time (before or after), and the interaction between the two significantly affected the numbers of crabs, with fewer crabs attracted to the chemical cue after noise exposure. The results indicate that noise can affect chemical information use in the marine environment, acting cross-modally to impact chemically-guided search behaviour in free-ranging animals. Broadly, anthropogenic noise and seabed vibration may have profound effects, even on behaviours mediated by other sensory modalities. Hence, the impact of noise should be investigated not only within, but also across sensory modalities.This article has an associated First Person interview with the first author of the paper.

| S-EPMC6679394 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data