Dataset Information

Crossmodal hierarchical predictive coding for audiovisual sequences in the human brain.

ABSTRACT: Predictive coding theory suggests the brain anticipates sensory information using prior knowledge. While this theory has been extensively researched within individual sensory modalities, evidence for predictive processing across sensory modalities is limited. Here, we examine how crossmodal knowledge is represented and learned in the brain, by identifying the hierarchical networks underlying crossmodal predictions when information of one sensory modality leads to a prediction in another modality. We record electroencephalogram (EEG) during a crossmodal audiovisual local-global oddball paradigm, in which the predictability of transitions between tones and images are manipulated at both the stimulus and sequence levels. To dissect the complex predictive signals in our EEG data, we employed a model-fitting approach to untangle neural interactions across modalities and hierarchies. The model-fitting result demonstrates that audiovisual integration occurs at both the levels of individual stimulus interactions and multi-stimulus sequences. Furthermore, we identify the spatio-spectro-temporal signatures of prediction-error signals across hierarchies and modalities, and reveal that auditory and visual prediction errors are rapidly redirected to the central-parietal electrodes during learning through alpha-band interactions. Our study suggests a crossmodal predictive coding mechanism where unimodal predictions are processed by distributed brain networks to form crossmodal knowledge.

SUBMITTER: Huang YT

PROVIDER: S-EPMC11316022 | biostudies-literature | 2024 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Crossmodal hierarchical predictive coding for audiovisual sequences in the human brain.

Huang Yiyuan Teresa YT Wu Chien-Te CT Fang Yi-Xin Miranda YM Fu Chin-Kun CK Koike Shinsuke S Chao Zenas C ZC

Communications biology 20240809 1

Predictive coding theory suggests the brain anticipates sensory information using prior knowledge. While this theory has been extensively researched within individual sensory modalities, evidence for predictive processing across sensory modalities is limited. Here, we examine how crossmodal knowledge is represented and learned in the brain, by identifying the hierarchical networks underlying crossmodal predictions when information of one sensory modality leads to a prediction in another modality ...[more]

PMID: 39122960

Similar Datasets

Project description:Right brain-damaged patients with unilateral spatial neglect fail to explore the left side of space. Recent EEG and clinical evidence suggests that neglect patients might suffer deficits in predictive coding, i.e. in identifying and exploiting probabilistic associations among sensory stimuli in the environment. To gain direct insights on this issue, we focussed on the hierarchical components of predictive coding. We recorded EEG responses evoked by central, left-side or right-side tones that were presented at the end of sequences of four central tones. Left-side and right-side deviant tones produce a pre-attentive Mismatch Negativity that reflects a lower-order prediction error for the 'Local' deviation of the tone at the end of the sequence. Higher-order prediction errors for the frequency of these deviations in the acoustic environment, i.e. 'Global' deviation, are marked by the P3 response. We show that when neglect patients are immersed in an acoustic environment characterized by frequent left-side deviant tones, they display no pre-attentive Mismatch Negativity both for left-side deviant tones and infrequent omissions of the last tone, while they have Mismatch Negativity for infrequent right-side deviant tones. In the same condition, neglect patients show no P300 response to 'Global' prediction errors for deviant tones, including those in the non-neglected right-side, and omissions. In contrast to this, when right-side deviant tones are predominant in the acoustic environment, neglect patients have pre-attentive Mismatch Negativity both for right-side deviant tones and infrequent omissions, while they display no Mismatch Negativity for infrequent left-side deviant tones. Most importantly, in the same condition neglect patients show enhanced P300 response to infrequent left-side deviant tones, notwithstanding that these tones evoked no pre-attentive Mismatch Negativity. This latter finding indicates that 'Global' predictions are independent of 'Local' error signals provided by the Mismatch Negativity. These results qualify deficits of predictive coding in the spatial neglect syndrome and show that neglect patients base their predictive behaviour only on statistical regularities that are related to the frequent occurrence of sensory events on the right side of space.

Project description:The visual color-word Stroop task is widely used in clinical and research settings as a measure of cognitive control. Numerous neuroimaging studies have used color-word Stroop tasks to investigate the neural resources supporting cognitive control, but to our knowledge all have used unimodal (typically visual) Stroop paradigms. Thus, it is possible that this classic measure of cognitive control is not capturing the resources involved in multisensory cognitive control. The audiovisual integration and crossmodal correspondence literatures identify regions sensitive to congruency of auditory and visual stimuli, but it is unclear how these regions relate to the unimodal cognitive control literature. In this study we aimed to identify brain regions engaged by crossmodal cognitive control during an audiovisual color-word Stroop task, and how they relate to previous unimodal Stroop and audiovisual integration findings. First, we replicated previous behavioral audiovisual Stroop findings in an fMRI-adapted audiovisual Stroop paradigm: incongruent visual information increased reaction time towards an auditory stimulus and congruent visual information decreased reaction time. Second, we investigated the brain regions supporting cognitive control during an audiovisual color-word Stroop task using fMRI. Similar to unimodal cognitive control tasks, a left superior parietal region exhibited an interference effect of visual information on the auditory stimulus. This superior parietal region was also identified using a standard audiovisual integration localizing procedure, indicating that audiovisual integration resources are sensitive to cognitive control demands. Facilitation of the auditory stimulus by congruent visual information was found in posterior superior temporal cortex, including in the posterior STS which has been found to support audiovisual integration. The dorsal anterior cingulate cortex, often implicated in unimodal Stroop tasks, was not modulated by the audiovisual Stroop task. Overall the findings indicate that an audiovisual color-word Stroop task engages overlapping resources with audiovisual integration and overlapping but distinct resources compared to unimodal Stroop tasks.

Dataset Information

Crossmodal hierarchical predictive coding for audiovisual sequences in the human brain.

Publications

Crossmodal hierarchical predictive coding for audiovisual sequences in the human brain.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets