Speaker-independent auditory attention decoding without access to clean speech sources.
Ontology highlight
ABSTRACT:
SUBMITTER: Han C
PROVIDER: S-EPMC6520028 | biostudies-literature | 2019 May
REPOSITORIES: biostudies-literature
Publications
Speaker-independent auditory attention decoding without access to clean speech sources.
Han Cong C O'Sullivan James J Luo Yi Y Herrero Jose J Mehta Ashesh D AD Mesgarani Nima N
Science advances 20190515 5
Speech perception in crowded environments is challenging for hearing-impaired listeners. Assistive hearing devices cannot lower interfering speakers without knowing which speaker the listener is focusing on. One possible solution is auditory attention decoding in which the brainwaves of listeners are compared with sound sources to determine the attended source, which can then be amplified to facilitate hearing. In realistic situations, however, only mixed audio is available. We utilize a novel s ...[more]
Similar Datasets
Project description:IntroductionIn our complex world, the auditory system plays a crucial role in perceiving and processing our environment. Humans are able to segment and stream concurrent auditory objects, allowing them to focus on specific sounds, such as speech, and suppress irrelevant auditory objects. The attentional enhancement or suppression of sound processing is evident in neural data through a phenomenon called neural speech tracking. Previous studies have identified correlates of neural speech tracking in electroencephalography (EEG) data, but EEG measures are susceptible to motion artefacts, and the association between neural data and auditory objects is vulnerable to distraction.MethodsThe current study investigated EEG-based auditory attention decoding in realistic everyday scenarios. N=20 participants were exposed to the sound of a busy cafeteria or walked along busy and quiet streets while listening to one or two simultaneous speech streams. We also investigated the robustness of neural speech tracking estimates within subjects. Linear decoding models were used to determine the magnitude of neural speech tracking.ResultsThe results confirmed that neural speech tracking was strongest in single speaker scenarios. In dual speaker conditions, there was significantly stronger neural speech tracking for the attended speaker compared to the ignored speaker, even in complex environments such as a busy cafeteria or outdoor settings.DiscussionIn conclusion, EEG-based attention decoding is feasible in highly complex and realistic everyday conditions while humans behave naturally.
| S-EPMC11599177 | biostudies-literature
Project description:Human listeners can focus on one speech stream out of several concurrent ones. The present study aimed to assess the whole-brain functional networks underlying a) the process of focusing attention on a single speech stream vs. dividing attention between two streams and 2) speech processing on different time-scales and depth. Two spoken narratives were presented simultaneously while listeners were instructed to a) track and memorize the contents of a speech stream and b) detect the presence of numerals or syntactic violations in the same ("focused attended condition") or in the parallel stream ("divided attended condition"). Speech content tracking was found to be associated with stronger connectivity in lower frequency bands (delta band- 0,5-4 Hz), whereas the detection tasks were linked with networks operating in the faster alpha (8-10 Hz) and beta (13-30 Hz) bands. These results suggest that the oscillation frequencies of the dominant brain networks during speech processing may be related to the duration of the time window within which information is integrated. We also found that focusing attention on a single speaker compared to dividing attention between two concurrent speakers was predominantly associated with connections involving the frontal cortices in the delta (0.5-4 Hz), alpha (8-10 Hz), and beta bands (13-30 Hz), whereas dividing attention between two parallel speech streams was linked with stronger connectivity involving the parietal cortices in the delta and beta frequency bands. Overall, connections strengthened by focused attention may reflect control over information selection, whereas connections strengthened by divided attention may reflect the need for maintaining two streams in parallel and the related control processes necessary for performing the tasks.
| S-EPMC6394951 | biostudies-literature
Project description:The recent progress in auditory attention decoding (AAD) methods is based on algorithms that find a relation between the audio envelope and the neurophysiological response. The most popular approach is based on the reconstruction of the audio envelope from electroencephalogram (EEG) signals. These methods are primarily based on the exogenous response driven by the physical characteristics of the stimuli. In this study, we specifically investigate higher-level cognitive responses influenced by auditory attention to natural speech events. We designed a series of four experimental paradigms with increasing levels of realism: a word category oddball paradigm, a word category oddball paradigm with competing speakers, and competing speech streams with and without specific targets. We recorded EEG data using 32 scalp electrodes, as well as 12 in-ear electrodes (ear-EEG) from 24 participants. By using natural speech events and cognitive tasks, a cognitive event-related potential (ERP) component, which we believe is related to the well-known P3b component, was observed at parietal electrode sites with a latency of ~625 ms. Importantly, the component decreases in strength but is still significantly observable in increasingly realistic paradigms of multi-talker environments. We also show that the component can be observed in the in-ear EEG signals by using spatial filtering. We believe that the P3b-like cognitive component modulated by auditory attention can contribute to improving auditory attention decoding from electrophysiological recordings.
| S-EPMC11743694 | biostudies-literature
Project description:Auditory attention is an important cognitive function used to separate relevant from irrelevant auditory information. However, most findings on attentional selection have been obtained in highly controlled laboratory settings using bulky recording setups and unnaturalistic stimuli. Recent advances in electroencephalography (EEG) facilitate the measurement of brain activity outside the laboratory, and around-the-ear sensors such as the cEEGrid promise unobtrusive acquisition. In parallel, methods such as speech envelope tracking, intersubject correlations and spectral entropy measures emerged which allow us to study attentional effects in the neural processing of natural, continuous auditory scenes. In the current study, we investigated whether these three attentional measures can be reliably obtained when using around-the-ear EEG. To this end, we analyzed the cEEGrid data of 36 participants who attended to one of two simultaneously presented speech streams. Speech envelope tracking results confirmed a reliable identification of the attended speaker from cEEGrid data. The accuracies in identifying the attended speaker increased when fitting the classification model to the individual. Artifact correction of the cEEGrid data with artifact subspace reconstruction did not increase the classification accuracy. Intersubject correlations were higher for those participants attending to the same speech stream than for those attending to different speech streams, replicating previously obtained results with high-density cap-EEG. We also found that spectral entropy decreased over time, possibly reflecting the decrease in the listener's level of attention. Overall, these results support the idea of using ear-EEG measurements to unobtrusively monitor auditory attention to continuous speech. This knowledge may help to develop assistive devices that support listeners separating relevant from irrelevant information in complex auditory environments.
| S-EPMC9111016 | biostudies-literature
Project description:Focusing attention on one speaker on the background of other irrelevant speech can be a challenging feat. A longstanding question in attention research is whether and how frequently individuals shift their attention towards task-irrelevant speech, arguably leading to occasional detection of words in a so-called unattended message. However, this has been difficult to gauge empirically, particularly when participants attend to continuous natural speech, due to the lack of appropriate metrics for detecting shifts in internal attention. Here we introduce a new experimental platform for studying the dynamic deployment of attention among concurrent speakers, utilizing a unique combination of Virtual Reality (VR) and Eye-Tracking technology. We created a Virtual Café in which participants sit across from and attend to the narrative of a target speaker. We manipulated the number and location of distractor speakers by placing additional characters throughout the Virtual Café. By monitoring participant's eye-gaze dynamics, we studied the patterns of overt attention-shifts among concurrent speakers as well as the consequences of these shifts on speech comprehension. Our results reveal important individual differences in the gaze-pattern displayed during selective attention to speech. While some participants stayed fixated on a target speaker throughout the entire experiment, approximately 30% of participants frequently shifted their gaze toward distractor speakers or other locations in the environment, regardless of the severity of audiovisual distraction. Critically, preforming frequent gaze-shifts negatively impacted the comprehension of target speech, and participants made more mistakes when looking away from the target speaker. We also found that gaze-shifts occurred primarily during gaps in the acoustic input, suggesting that momentary reductions in acoustic masking prompt attention-shifts between competing speakers, in line with "glimpsing" theories of processing speech in noise. These results open a new window into understanding the dynamics of attention as they wax and wane over time, and the different listening patterns employed for dealing with the influx of sensory input in multisensory environments. Moreover, the novel approach developed here for tracking the locus of momentary attention in a naturalistic virtual-reality environment holds high promise for extending the study of human behavior and cognition and bridging the gap between the laboratory and real-life.
| S-EPMC6857110 | biostudies-literature
Project description:Rapid recognition and categorization of sounds are essential for humans and animals alike, both for understanding and reacting to our surroundings and for daily communication and social interaction. For humans, perception of speech sounds is of crucial importance. In real life, this task is complicated by the presence of a multitude of meaningful non-speech sounds. The present behavioural, magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI) study was set out to address how attention to speech versus attention to natural non-speech sounds within complex auditory scenes influences cortical processing. The stimuli were superimpositions of spoken words and environmental sounds, with parametric variation of the speech-to-environmental sound intensity ratio. The participants' task was to detect a repetition in either the speech or the environmental sound. We found that specifically when participants attended to speech within the superimposed stimuli, higher speech-to-environmental sound ratios resulted in shorter sustained MEG responses and stronger BOLD fMRI signals especially in the left supratemporal auditory cortex and in improved behavioural performance. No such effects of speech-to-environmental sound ratio were observed when participants attended to the environmental sound part within the exact same stimuli. These findings suggest stronger saliency of speech compared with other meaningful sounds during processing of natural auditory scenes, likely linked to speech-specific top-down and bottom-up mechanisms activated during speech perception that are needed for tracking speech in real-life-like auditory environments.
| S-EPMC9298413 | biostudies-literature
Project description:Human brain performs remarkably well in segregating a particular speaker from interfering ones in a multispeaker scenario. We can quantitatively evaluate the segregation capability by modeling a relationship between the speech signals present in an auditory scene, and the listener's cortical signals measured using electroencephalography (EEG). This has opened up avenues to integrate neuro-feedback into hearing aids where the device can infer user's attention and enhance the attended speaker. Commonly used algorithms to infer the auditory attention are based on linear systems theory where cues such as speech envelopes are mapped on to the EEG signals. Here, we present a joint convolutional neural network (CNN)-long short-term memory (LSTM) model to infer the auditory attention. Our joint CNN-LSTM model takes the EEG signals and the spectrogram of the multiple speakers as inputs and classifies the attention to one of the speakers. We evaluated the reliability of our network using three different datasets comprising of 61 subjects, where each subject undertook a dual-speaker experiment. The three datasets analyzed corresponded to speech stimuli presented in three different languages namely German, Danish, and Dutch. Using the proposed joint CNN-LSTM model, we obtained a median decoding accuracy of 77.2% at a trial duration of 3 s. Furthermore, we evaluated the amount of sparsity that the model can tolerate by means of magnitude pruning and found a tolerance of up to 50% sparsity without substantial loss of decoding accuracy.
| S-EPMC8365753 | biostudies-literature
Project description:Auditory verbal hallucinations (AVH) are a hallmark of psychotic experience. Various mechanisms including misattribution of inner speech and imbalance between bottom-up and top-down factors in auditory perception potentially due to aberrant connectivity between frontal and temporo-parietal areas have been suggested to underlie AVH. Experimental evidence for disturbed connectivity of networks sustaining auditory-verbal processing is, however, sparse. We compared functional resting-state connectivity in 49 psychotic patients with frequent AVH and 49 matched controls. The analysis was seeded from the left middle temporal gyrus (MTG), thalamus, angular gyrus (AG) and inferior frontal gyrus (IFG) as these regions are implicated in extracting meaning from impoverished speech-like sounds. Aberrant connectivity was found for all seeds. Decreased connectivity was observed between the left MTG and its right homotope, between the left AG and the surrounding inferior parietal cortex (IPC) and the left inferior temporal gyrus, between the left thalamus and the right cerebellum, as well as between the left IFG and left IPC, and dorsolateral and ventrolateral prefrontal cortex (DLPFC/VLPFC). Increased connectivity was observed between the left IFG and the supplementary motor area (SMA) and the left insula and between the left thalamus and the left fusiform gyrus/hippocampus. The predisposition to experience AVH might result from decoupling between the speech production system (IFG, insula and SMA) and the self-monitoring system (DLPFC, VLPFC, IPC) leading to misattribution of inner speech. Furthermore, decreased connectivity between nodes involved in speech processing (AG, MTG) and other regions implicated in auditory processing might reflect aberrant top-down influences in AVH.
| S-EPMC8011946 | biostudies-literature
Project description:Human listeners have the ability to direct their attention to a single speaker in a multi-talker environment. The neural correlates of selective attention can be decoded from a single trial of electroencephalography (EEG) data. In this study, leveraging the source-reconstructed and anatomically-resolved EEG data as inputs, we sought to employ CNN as an interpretable model to uncover task-specific interactions between brain regions, rather than simply to utilize it as a black box decoder. To this end, our CNN model was specifically designed to learn pairwise interaction representations for 10 cortical regions from five-second inputs. By exclusively utilizing these features for decoding, our model was able to attain a median accuracy of 77.56% for within-participant and 65.14% for cross-participant classification. Through ablation analysis together with dissecting the features of the models and applying cluster analysis, we were able to discern the presence of alpha-band-dominated inter-hemisphere interactions, as well as alpha- and beta-band dominant interactions that were either hemisphere-specific or were characterized by a contrasting pattern between the right and left hemispheres. These interactions were more pronounced in parietal and central regions for within-participant decoding, but in parietal, central, and partly frontal regions for cross-participant decoding. These findings demonstrate that our CNN model can effectively utilize features known to be important in auditory attention tasks and suggest that the application of domain knowledge inspired CNNs on source-reconstructed EEG data can offer a novel computational framework for studying task-relevant brain interactions.
| S-EPMC11335149 | biostudies-literature
Project description:ObjectiveTo assess whether speech therapy can lead to better results for early cochlear implantation (CI) children.PatientsA cohort of thirty-four congenitally profoundly deaf children who underwent CI before the age of 18 months at the Sixth Hospital affiliated with Shanghai Jiaotong University from January 2005 to July 2008 were included. Nineteen children received speech therapy in rehabilitation centers (ST), whereas the remaining fifteen cases did not (NST), but were exposed to the real world, as are normal hearing children.MethodsAll children were assessed before surgery and at 6, 12, and 24 months after surgery with the Categories of Auditory Performance test (CAP) and the Speech Intelligibility Rating (SIR). Each assessment was given by the same therapist who was blind to the situation of the child at each observation interval. CAP and SIR scores of the groups were compared at each time point.ResultsOur study showed that the auditory performance and speech intelligibility of trained children were almost the same as to those of untrained children with early implantation. The CAP and SIR scores of both groups increased with increased time of implant use during the follow-up period, and at each time point, the median scores of the two groups were about equal.ConclusionsThese results indicate that great communication benefits are achieved by early implantation (<18 months) without routine speech therapy. The results exemplify the importance of enhanced social environments provided by everyday life experience for human brain development and reassure parents considering cochlear implants where speech training is unavailable.
| S-EPMC3549925 | biostudies-literature