Dataset Information

Lip-reading aids word recognition most in moderate noise: a Bayesian explanation using high-dimensional feature space.

ABSTRACT: Watching a speaker's facial movements can dramatically enhance our ability to comprehend words, especially in noisy environments. From a general doctrine of combining information from different sensory modalities (the principle of inverse effectiveness), one would expect that the visual signals would be most effective at the highest levels of auditory noise. In contrast, we find, in accord with a recent paper, that visual information improves performance more at intermediate levels of auditory noise than at the highest levels, and we show that a novel visual stimulus containing only temporal information does the same. We present a Bayesian model of optimal cue integration that can explain these conflicts. In this model, words are regarded as points in a multidimensional space and word recognition is a probabilistic inference process. When the dimensionality of the feature space is low, the Bayesian model predicts inverse effectiveness; when the dimensionality is high, the enhancement is maximal at intermediate auditory noise levels. When the auditory and visual stimuli differ slightly in high noise, the model makes a counterintuitive prediction: as sound quality increases, the proportion of reported words corresponding to the visual stimulus should first increase and then decrease. We confirm this prediction in a behavioral experiment. We conclude that auditory-visual speech perception obeys the same notion of optimality previously observed only for simple multisensory stimuli.

SUBMITTER: Ma WJ

PROVIDER: S-EPMC2645675 | biostudies-literature | 2009

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Lip-reading aids word recognition most in moderate noise: a Bayesian explanation using high-dimensional feature space.

Ma Wei Ji WJ Zhou Xiang X Ross Lars A LA Foxe John J JJ Parra Lucas C LC

PloS one 20090304 3

Watching a speaker's facial movements can dramatically enhance our ability to comprehend words, especially in noisy environments. From a general doctrine of combining information from different sensory modalities (the principle of inverse effectiveness), one would expect that the visual signals would be most effective at the highest levels of auditory noise. In contrast, we find, in accord with a recent paper, that visual information improves performance more at intermediate levels of auditory n ...[more]

PMID: 19259259

Similar Datasets

Project description:For normally sighted readers, word neighborhood size (i.e., the total number of words that can be formed from a single word by changing only one letter) has a facilitator effect on word recognition. When reading with central field loss (CFL) however, individual letters may not be correctly identified, leading to possible misidentifications and a reverse neighborhood size effect. Here we investigate this inhibitory effect of word neighborhood size on reading performance and whether it is modulated by word predictability and reading proficiency. Nineteen patients with binocular CFL from 32 to 89 years old (mean ± SD = 75 ± 15) read short sentences presented with the self-paced reading paradigm. Accuracy and reading time were measured for each target word read, along with its predictability, i.e., its probability of occurrence following the two preceding words in the sentence using a trigram analysis. Linear mixed effects models were then fit to estimate the individual contributions of word neighborhood size, predictability, frequency and length on accuracy and reading time, while taking patients' reading proficiency into account. For the less proficient readers, who have given up daily reading as a consequence of their visual impairment, we found that the effect of neighborhood size was reversed compared to normally sighted readers and of higher amplitude than the effect of frequency. Furthermore, this inhibitory effect is of greater amplitude (up to 50% decrease in reading speed) when a word is not easily predictable because its chances to occur after the two preceding words in a specific sentence are rather low. Severely impaired patients with CFL often quit reading on a daily basis because this task becomes simply too exhausting. Based on our results, we envision lexical text simplification as a new alternative to promote effective rehabilitation in these patients. By increasing reading accessibility for those who struggle the most, text simplification might be used as an efficient rehabilitation tool and daily reading assistive technology, fostering overall reading ability and fluency through increased practice.

Dataset Information

Lip-reading aids word recognition most in moderate noise: a Bayesian explanation using high-dimensional feature space.

Publications

Lip-reading aids word recognition most in moderate noise: a Bayesian explanation using high-dimensional feature space.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets