Browse
Submit Data
Databases
API
Help

Dataset Information

0 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Studying naturalistic human communication using dual-EEG and audio-visual recordings.

ABSTRACT: We present a protocol to study naturalistic human communication using dual-electroencephalography (EEG) and audio-visual recordings. We describe preparatory steps for data collection including setup preparation, experiment design, and piloting. We then describe the data collection process in detail which consists of participant recruitment, experiment room preparation, and data collection. We also outline the kinds of research questions that can be addressed with the current protocol, including several analysis possibilities, from conversational to advanced time-frequency analyses. For complete details on the use and execution of this protocol, please refer to Drijvers and Holler (2022).¹.

SUBMITTER: Mazzini S

PROVIDER: S-EPMC10511849 | biostudies-literature | 2023 Jul

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Publications

Studying naturalistic human communication using dual-EEG and audio-visual recordings.

Mazzini Sara S Holler Judith J Drijvers Linda L

STAR protocols 20230707 3

We present a protocol to study naturalistic human communication using dual-electroencephalography (EEG) and audio-visual recordings. We describe preparatory steps for data collection including setup preparation, experiment design, and piloting. We then describe the data collection process in detail which consists of participant recruitment, experiment room preparation, and data collection. We also outline the kinds of research questions that can be addressed with the current protocol, including ...[more]

PMID: 37421617

Similar Datasets

Semi-Transparent, Pixel-Free Upconversion Goggles with Dual Audio-Visual Communication.

Project description:The intractable brittleness and opacity of the crystalline semiconductor restrict the prospect of developing low-cost imaging systems. Here, infrared visualization technologies are established with large-area, semi-transparent organic upconversion devices that bring high-resolution invisible images into sight without photolithography. To exploit all photoinduced charge carriers, a monolithic device structure is proposed built on the infrared-selective, single-component charge generation layer of chloroaluminum phthalocyanine (ClAlPc) coupled to two visible light-emitting layers manipulated with unipolar charges. Transient pump-probe spectroscopy reveals that the ClAlPc-based device exhibits an efficient charge dissociation process under forward bias. This process is indicated by the prompt and strong features of electroabsorption screening. Furthermore, by imposing the electric field, the ultrafast excited state dynamic suggests a prolonged charge carrier lifetime from the ClAlPc, which facilitates the charge utilization for upconversion luminance. For the first time, >30% of the infrared photons are utilized without photomultiplication strategies owing to the trivial spectrum overlap between ClAlPc and the emitter. In addition, the device can broadcast the acoustic signal by synchronizing the device frequency with the light source, which enables to operate it in dual audio-visual mode. The work demonstrates the potential of upconversion devices for affordable infrared imaging in wearable electronics.

| S-EPMC10625064 | biostudies-literature

Unsupervised Decoding of Long-Term, Naturalistic Human Neural Recordings with Automated Video and Audio Annotations.

Project description:Fully automated decoding of human activities and intentions from direct neural recordings is a tantalizing challenge in brain-computer interfacing. Implementing Brain Computer Interfaces (BCIs) outside carefully controlled experiments in laboratory settings requires adaptive and scalable strategies with minimal supervision. Here we describe an unsupervised approach to decoding neural states from naturalistic human brain recordings. We analyzed continuous, long-term electrocorticography (ECoG) data recorded over many days from the brain of subjects in a hospital room, with simultaneous audio and video recordings. We discovered coherent clusters in high-dimensional ECoG recordings using hierarchical clustering and automatically annotated them using speech and movement labels extracted from audio and video. To our knowledge, this represents the first time techniques from computer vision and speech processing have been used for natural ECoG decoding. Interpretable behaviors were decoded from ECoG data, including moving, speaking and resting; the results were assessed by comparison with manual annotation. Discovered clusters were projected back onto the brain revealing features consistent with known functional areas, opening the door to automated functional brain mapping in natural settings.

| S-EPMC4838634 | biostudies-literature

Human EEG recordings for 1,854 concepts presented in rapid serial visual presentation streams.

Project description:The neural basis of object recognition and semantic knowledge has been extensively studied but the high dimensionality of object space makes it challenging to develop overarching theories on how the brain organises object knowledge. To help understand how the brain allows us to recognise, categorise, and represent objects and object categories, there is a growing interest in using large-scale image databases for neuroimaging experiments. In the current paper, we present THINGS-EEG, a dataset containing human electroencephalography responses from 50 subjects to 1,854 object concepts and 22,248 images in the THINGS stimulus set, a manually curated and high-quality image database that was specifically designed for studying human vision. The THINGS-EEG dataset provides neuroimaging recordings to a systematic collection of objects and concepts and can therefore support a wide array of research to understand visual object processing in the human brain.

| S-EPMC8748587 | biostudies-literature

Animated virtual characters to explore audio-visual speech in controlled and naturalistic environments.

Project description:Natural speech is processed in the brain as a mixture of auditory and visual features. An example of the importance of visual speech is the McGurk effect and related perceptual illusions that result from mismatching auditory and visual syllables. Although the McGurk effect has widely been applied to the exploration of audio-visual speech processing, it relies on isolated syllables, which severely limits the conclusions that can be drawn from the paradigm. In addition, the extreme variability and the quality of the stimuli usually employed prevents comparability across studies. To overcome these limitations, we present an innovative methodology using 3D virtual characters with realistic lip movements synchronized on computer-synthesized speech. We used commercially accessible and affordable tools to facilitate reproducibility and comparability, and the set-up was validated on 24 participants performing a perception task. Within complete and meaningful French sentences, we paired a labiodental fricative viseme (i.e. /v/) with a bilabial occlusive phoneme (i.e. /b/). This audiovisual mismatch is known to induce the illusion of hearing /v/ in a proportion of trials. We tested the rate of the illusion while varying the magnitude of background noise and audiovisual lag. Overall, the effect was observed in 40% of trials. The proportion rose to about 50% with added background noise and up to 66% when controlling for phonetic features. Our results conclusively demonstrate that computer-generated speech stimuli are judicious, and that they can supplement natural speech with higher control over stimulus timing and content.

| S-EPMC7511320 | biostudies-literature

A studyforrest extension, MEG recordings while watching the audio-visual movie "Forrest Gump".

Project description:Naturalistic stimuli, such as movies, are being increasingly used to map brain function because of their high ecological validity. The pioneering studyforrest and other naturalistic neuroimaging projects have provided free access to multiple movie-watching functional magnetic resonance imaging (fMRI) datasets to prompt the community for naturalistic experimental paradigms. However, sluggish blood-oxygenation-level-dependent fMRI signals are incapable of resolving neuronal activity with the temporal resolution at which it unfolds. Instead, magnetoencephalography (MEG) measures changes in the magnetic field produced by neuronal activity and is able to capture rich dynamics of the brain at the millisecond level while watching naturalistic movies. Herein, we present the first public prolonged MEG dataset collected from 11 participants while watching the 2 h long audio-visual movie "Forrest Gump". Minimally preprocessed data was also provided to facilitate the use of the dataset. As a studyforrest extension, we envision that this dataset, together with fMRI data from the studyforrest project, will serve as a foundation for exploring the neural dynamics of various cognitive functions in real-world contexts.

| S-EPMC9106652 | biostudies-literature

Studying memory processes at different levels with simultaneous depth and surface EEG recordings.

Project description:Investigating cognitive brain functions using non-invasive electrophysiology can be challenging due to the particularities of the task-related EEG activity, the depth of the activated brain areas, and the extent of the networks involved. Stereoelectroencephalographic (SEEG) investigations in patients with drug-resistant epilepsy offer an extraordinary opportunity to validate information derived from non-invasive recordings at macro-scales. The SEEG approach can provide brain activity with high spatial specificity during tasks that target specific cognitive processes (e.g., memory). Full validation is possible only when performing simultaneous scalp SEEG recordings, which allows recording signals in the exact same brain state. This is the approach we have taken in 12 subjects performing a visual memory task that requires the recognition of previously viewed objects. The intracranial signals on 965 contact pairs have been compared to 391 simultaneously recorded scalp signals at a regional and whole-brain level, using multivariate pattern analysis. The results show that the task conditions are best captured by intracranial sensors, despite the limited spatial coverage of SEEG electrodes, compared to the whole-brain non-invasive recordings. Applying beamformer source reconstruction or independent component analysis does not result in an improvement of the multivariate task decoding performance using surface sensor data. By analyzing a joint scalp and SEEG dataset, we investigated whether the two types of signals carry complementary information that might improve the machine-learning classifier performance. This joint analysis revealed that the results are driven by the modality exhibiting best individual performance, namely SEEG.

| S-EPMC10110965 | biostudies-literature

Naturalistic Audio-Movies reveal common spatial organization across "visual" cortices of different blind individuals.

Project description:Occipital cortices of different sighted people contain analogous maps of visual information (e.g. foveal vs. peripheral). In congenital blindness, "visual" cortices respond to nonvisual stimuli. Do visual cortices of different blind people represent common informational maps? We leverage naturalistic stimuli and inter-subject pattern similarity analysis to address this question. Blindfolded sighted (n = 22) and congenitally blind (n = 22) participants listened to 6 sound clips (5-7 min each): 3 auditory excerpts from movies; a naturalistic spoken narrative; and matched degraded auditory stimuli (Backwards Speech, scrambled sentences), during functional magnetic resonance imaging scanning. We compared the spatial activity patterns evoked by each unique 10-s segment of the different auditory excerpts across blind and sighted people. Segments of meaningful naturalistic stimuli produced distinctive activity patterns in frontotemporal networks that were shared across blind and across sighted individuals. In the blind group only, segment-specific, cross-subject patterns emerged in visual cortex, but only for meaningful naturalistic stimuli and not Backwards Speech. Spatial patterns of activity within visual cortices are sensitive to time-varying information in meaningful naturalistic auditory stimuli in a broadly similar manner across blind individuals.

| S-EPMC9758574 | biostudies-literature

Investigating the attentional focus to workplace-related soundscapes in a complex audio-visual-motor task using EEG.

Project description:IntroductionIn demanding work situations (e.g., during a surgery), the processing of complex soundscapes varies over time and can be a burden for medical personnel. Here we study, using mobile electroencephalography (EEG), how humans process workplace-related soundscapes while performing a complex audio-visual-motor task (3D Tetris). Specifically, we wanted to know how the attentional focus changes the processing of the soundscape as a whole.MethodParticipants played a game of 3D Tetris in which they had to use both hands to control falling blocks. At the same time, participants listened to a complex soundscape, similar to what is found in an operating room (i.e., the sound of machinery, people talking in the background, alarm sounds, and instructions). In this within-subject design, participants had to react to instructions (e.g., "place the next block in the upper left corner") and to sounds depending on the experimental condition, either to a specific alarm sound originating from a fixed location or to a beep sound that originated from varying locations. Attention to the alarm reflected a narrow attentional focus, as it was easy to detect and most of the soundscape could be ignored. Attention to the beep reflected a wide attentional focus, as it required the participants to monitor multiple different sound streams.Results and discussionResults show the robustness of the N1 and P3 event related potential response during this dynamic task with a complex auditory soundscape. Furthermore, we used temporal response functions to study auditory processing to the whole soundscape. This work is a step toward studying workplace-related sound processing in the operating room using mobile EEG.

| S-EPMC10790850 | biostudies-literature

Using naturalistic recordings to study children's social perceptions and evaluations.

Project description:Children often encounter events that bear on their moral and other evaluations, such as physical aggression and material disorder. Children's perceptions and evaluations are decisive for how they respond to and learn from these everyday events. Using a new method for investigating the development of social perceptions and evaluations, researchers interviewed 3- to 6-year-olds about naturalistic video recordings of harm, disorder, and joint play events. Children distinguished between the situations in the perceived intent of the protagonists, evaluations, justifications, and what they thought the protagonist should do after. Age differences suggested that perceiving and evaluating simple everyday situations was challenging for younger children. This research highlights the importance of studying children's everyday social perceptions and evaluations and validates a new paradigm for doing so. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

| S-EPMC6698909 | biostudies-literature

AJILE12: Long-term naturalistic human intracranial neural recordings and pose.

Project description:Understanding the neural basis of human movement in naturalistic scenarios is critical for expanding neuroscience research beyond constrained laboratory paradigms. Here, we describe our Annotated Joints in Long-term Electrocorticography for 12 human participants (AJILE12) dataset, the largest human neurobehavioral dataset that is publicly available; the dataset was recorded opportunistically during passive clinical epilepsy monitoring. AJILE12 includes synchronized intracranial neural recordings and upper body pose trajectories across 55 semi-continuous days of naturalistic movements, along with relevant metadata, including thousands of wrist movement events and annotated behavioral states. Neural recordings are available at 500 Hz from at least 64 electrodes per participant, for a total of 1280 hours. Pose trajectories at 9 upper-body keypoints were estimated from 118 million video frames. To facilitate data exploration and reuse, we have shared AJILE12 on The DANDI Archive in the Neurodata Without Borders (NWB) data standard and developed a browser-based dashboard.

| S-EPMC9023453 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data