Dataset Information

Simulation of talking faces in the human brain improves auditory speech recognition.

ABSTRACT: Human face-to-face communication is essentially audiovisual. Typically, people talk to us face-to-face, providing concurrent auditory and visual input. Understanding someone is easier when there is visual input, because visual cues like mouth and tongue movements provide complementary information about speech content. Here, we hypothesized that, even in the absence of visual input, the brain optimizes both auditory-only speech and speaker recognition by harvesting speaker-specific predictions and constraints from distinct visual face-processing areas. To test this hypothesis, we performed behavioral and neuroimaging experiments in two groups: subjects with a face recognition deficit (prosopagnosia) and matched controls. The results show that observing a specific person talking for 2 min improves subsequent auditory-only speech and speaker recognition for this person. In both prosopagnosics and controls, behavioral improvement in auditory-only speech recognition was based on an area typically involved in face-movement processing. Improvement in speaker recognition was only present in controls and was based on an area involved in face-identity processing. These findings challenge current unisensory models of speech processing, because they show that, in auditory-only speech, the brain exploits previously encoded audiovisual correlations to optimize communication. We suggest that this optimization is based on speaker-specific audiovisual internal models, which are used to simulate a talking face.

SUBMITTER: von Kriegstein K

PROVIDER: S-EPMC2365564 | biostudies-literature | 2008 May

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Simulation of talking faces in the human brain improves auditory speech recognition.

von Kriegstein Katharina K Dogan Ozgür O Grüter Martina M Giraud Anne-Lise AL Kell Christian A CA Grüter Thomas T Kleinschmidt Andreas A Kiebel Stefan J SJ

Proceedings of the National Academy of Sciences of the United States of America 20080424 18

Human face-to-face communication is essentially audiovisual. Typically, people talk to us face-to-face, providing concurrent auditory and visual input. Understanding someone is easier when there is visual input, because visual cues like mouth and tongue movements provide complementary information about speech content. Here, we hypothesized that, even in the absence of visual input, the brain optimizes both auditory-only speech and speaker recognition by harvesting speaker-specific predictions an ...[more]

PMID: 18436648

Similar Datasets

Project description:ObjectivesPeople with Schizophrenia (SZ) show deficits in auditory and audiovisual speech recognition. It is possible that these deficits are related to aberrant early sensory processing, combined with an impaired ability to utilize visual cues to improve speech recognition. In this electroencephalography study we tested this by having SZ and healthy controls (HC) identify different unisensory auditory and bisensory audiovisual syllables at different auditory noise levels.MethodsSZ (N = 24) and HC (N = 21) identified one of three different syllables (/da/, /ga/, /ta/) at three different noise levels (no, low, high). Half the trials were unisensory auditory and the other half provided additional visual input of moving lips. Task-evoked mediofrontal N1 and P2 brain potentials triggered to the onset of the auditory syllables were derived and related to behavioral performance.ResultsIn comparison to HC, SZ showed speech recognition deficits for unisensory and bisensory stimuli. These deficits were primarily found in the no noise condition. Paralleling these observations, reduced N1 amplitudes to unisensory and bisensory stimuli in SZ were found in the no noise condition. In HC the N1 amplitudes were positively related to the speech recognition performance, whereas no such relationships were found in SZ. Moreover, no group differences in multisensory speech recognition benefits and N1 suppression effects for bisensory stimuli were observed.ConclusionOur study suggests that reduced N1 amplitudes reflect early auditory and audiovisual speech processing deficits in SZ. The findings that the amplitude effects were confined to salient speech stimuli and the attenuated relationship with behavioral performance in patients compared to HC, indicates a diminished decoding of the auditory speech signals in SZs. Our study also revealed relatively intact multisensory benefits in SZs, which implies that the observed auditory and audiovisual speech recognition deficits were primarily related to aberrant processing of the auditory syllables.

Project description:ObjectivesEveryday listening environments are filled with competing noise and distractors. Although significant research has examined the effect of competing noise on speech recognition and listening effort, little is understood about the effect of distraction. The framework for understanding effortful listening recognizes the importance of attention-related processes in speech recognition and listening effort; however, it underspecifies the role that they play, particularly with respect to distraction. The load theory of attention predicts that resources will be automatically allocated to processing a distractor, but only if perceptual load in the listening task is low enough. If perceptual load is high (i.e., listening in noise), then resources that would otherwise be allocated to processing a distractor are used to overcome the increased perceptual load and are unavailable for distractor processing. Although there is ample evidence for this theory in the visual domain, there has been little research investigating how the load theory of attention may apply to speech processing. In this study, we sought to measure the effect of distractors on speech recognition and listening effort and to evaluate whether the load theory of attention can be used to understand a listener's resource allocation in the presence of distractors.DesignFifteen adult listeners participated in a monosyllabic words repetition task. Test stimuli were presented in quiet or in competing speech (+5 dB signal-to-noise ratio) and in distractor or no distractor conditions. In conditions with distractors, auditory distractors were presented before the target words on 24% of the trials in quiet and in noise. Percent-correct was recorded as speech recognition, and verbal response time (VRT) was recorded as a measure of listening effort.ResultsA significant interaction was present for speech recognition, showing reduced speech recognition when distractors were presented in the quiet condition but no effect of distractors when noise was present. VRTs were significantly longer when distractors were present, regardless of listening condition.ConclusionsConsistent with the load theory of attention, distractors significantly reduced speech recognition in the low-perceptual load condition (i.e., listening in quiet) but did not impact speech recognition scores in conditions of high perceptual load (i.e., listening in noise). The increases in VRTs in the presence of distractors in both low- and high-perceptual load conditions (i.e., quiet and noise) suggest that the load theory of attention may not apply to listening effort. However, the large effect of distractors on VRT in both conditions is consistent with the previous work demonstrating that distraction-related shifts of attention can delay processing of the target task. These findings also fit within the framework for understanding effortful listening, which proposes that involuntary attentional shifts result in a depletion of cognitive resources, leaving less resources readily available to process the signal of interest; resulting in increased listening effort (i.e., elongated VRT).

Project description:The quality of the animal-human relationship and, consequently, the welfare of animals can be improved by gentle interactions such as stroking and talking. The perception of different stimuli during these interactions likely plays a key role in their emotional experience, but studies are scarce. During experiments, the standardization of verbal stimuli could be increased by using a recording. However, the use of a playback might influence the perception differently than "live" talking, which is closer to on-farm practice. Thus, we compared heifers' (n = 28) reactions to stroking while an experimenter was talking soothingly ("live") or while a recording of the experimenter talking soothingly was played ("playback"). Each animal was tested three times per condition and each trial comprised three phases: pre-stimulus, stimulus (stroking and talking) and post-stimulus. In both conditions, similar phrases with positive content were spoken calmly, using long low-pitched vowels. All tests were video recorded and analyzed for behaviors associated with different affective states. Effects on the heifers' cardiac parameters were assessed using analysis of heart rate variability. Independently of the auditory stimuli, longer durations of neck stretching occurred during stroking, supporting our hypothesis of a positive perception of stroking. Observation of ear positions revealed longer durations of the "back up" position and less ear flicking and changes of ear positions during stroking. The predicted decrease in HR during stroking was not confirmed; instead we found a slightly increased mean HR during stroking with a subsequent decrease in HR, which was stronger after stroking with live talking. In combination with differences in HRV parameters, our findings suggest that live talking might have been more pleasurable to the animals and had a stronger relaxing effect than "playback." The results regarding the effects of the degree of standardization of the stimulus on the variability of the data were inconclusive. We thus conclude that the use of recorded auditory stimuli to promote positive affective states during human-animal interactions in experimental settings is possible, but not necessarily preferable.

Dataset Information

Simulation of talking faces in the human brain improves auditory speech recognition.

Publications

Simulation of talking faces in the human brain improves auditory speech recognition.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets