Correlated lip motion and voice audio data.
Ontology highlight
ABSTRACT:
SUBMITTER: Colasito M
PROVIDER: S-EPMC6218630 | biostudies-literature | 2018 Dec
REPOSITORIES: biostudies-literature
Publications
Correlated lip motion and voice audio data.
Colasito Marco M Straub Jeremy J Kotala Pratap P
Data in brief 20181018
This data set is comprised of correlated audio and lip movement data in multiple videos of multiple subjects reading the same text. It was collected to facilitate the development and validation of algorithms used to train and test a compound biometric system that consists of lip-motion and voice recognition. The data set is a collection of videos of volunteers reciting a fixed script that is intended to be used to train software to recognize voice and lip-motion patterns. A second video is inclu ...[more]
Similar Datasets
Project description:Experimental research has shown that pairs of stimuli which are congruent and assumed to 'go together' are recalled more effectively than an item presented in isolation. Will this multisensory memory benefit occur when stimuli are richer and longer, in an ecological setting? In the present study, we focused on an everyday situation of audio-visual learning and manipulated the relationship between audio guide tracks and viewed portraits in the galleries of the Tate Britain. By varying the gender and narrative style of the voice-over, we examined how the perceived congruency and assumed unity of the audio guide track with painted portraits affected subsequent recall. We show that tracks perceived as best matching the viewed portraits led to greater recall of both sensory and linguistic content. We provide the first evidence that manipulating crossmodal congruence and unity assumptions can effectively impact memory in a multisensory ecological setting, even in the absence of precise temporal alignment between sensory cues.
| S-EPMC5479534 | biostudies-literature
Project description:The effects of asynchrony between audio and visual (A/V) stimuli on the N100m responses of magnetoencephalography in the left hemisphere were compared with those on the psychophysical responses in 11 participants. The latency and amplitude of N100m were significantly shortened and reduced in the left hemisphere by the presentation of visual speech as long as the temporal asynchrony between A/V stimuli was within 100 ms, but were not significantly affected with audio lags of -500 and +500 ms. However, some small effects were still preserved on average with audio lags of 500 ms, suggesting similar asymmetry of the temporal window to that observed in psychophysical measurements, which tended to be more robust (wider) for audio lags; i.e., the pattern of visual-speech effects as a function of A/V lag observed in the N100m in the left hemisphere grossly resembled that in psychophysical measurements on average, although the individual responses were somewhat varied. The present results suggest that the basic configuration of the temporal window of visual effects on auditory-speech perception could be observed from the early auditory processing stage.
| S-EPMC5193434 | biostudies-literature
Project description:Ultrasound is the primary modality for obstetric imaging and is highly sonographer dependent. Long training period, insufficient recruitment and poor retention of sonographers are among the global challenges in the expansion of ultrasound use. For the past several decades, technical advancements in clinical obstetric ultrasound scanning have largely concerned improving image quality and processing speed. By contrast, sonographers have been acquiring ultrasound images in a similar fashion for several decades. The PULSE (Perception Ultrasound by Learning Sonographer Experience) project is an interdisciplinary multi-modal imaging study aiming to offer clinical sonography insights and transform the process of obstetric ultrasound acquisition and image analysis by applying deep learning to large-scale multi-modal clinical data. A key novelty of the study is that we record full-length ultrasound video with concurrent tracking of the sonographer's eyes, voice and the transducer while performing routine obstetric scans on pregnant women. We provide a detailed description of the novel acquisition system and illustrate how our data can be used to describe clinical ultrasound. Being able to measure different sonographer actions or model tasks will lead to a better understanding of several topics including how to effectively train new sonographers, monitor the learning progress, and enhance the scanning workflow of experts.
| S-EPMC8266837 | biostudies-literature
Project description:Motor-task functional magnetic resonance imaging (fMRI) is crucial in the study of several clinical conditions, including stroke and Parkinson's disease. However, motor-task fMRI is complicated by task-correlated head motion, which can be magnified in clinical populations and confounds motor activation results. One method that may mitigate this issue is multi-echo independent component analysis (ME-ICA), which has been shown to separate the effects of head motion from the desired BOLD signal but has not been tested in motor-task datasets with high amounts of motion. In this study, we collected an fMRI dataset from a healthy population who performed a hand grasp task with and without task-correlated amplified head motion to simulate a motor-impaired population. We analyzed these data using three models: single-echo (SE), multi-echo optimally combined (ME-OC), and ME-ICA. We compared the models' performance in mitigating the effects of head motion on the subject level and group level. On the subject level, ME-ICA better dissociated the effects of head motion from the BOLD signal and reduced noise. Both ME models led to increased t-statistics in brain motor regions. In scans with high levels of motion, ME-ICA additionally mitigated artifacts and increased stability of beta coefficient estimates, compared to SE. On the group level, all three models produced activation clusters in expected motor areas in scans with both low and high motion, indicating that group-level averaging may also sufficiently resolve motion artifacts that vary by subject. These findings demonstrate that ME-ICA is a useful tool for subject-level analysis of motor-task data with high levels of task-correlated head motion. The improvements afforded by ME-ICA are critical to improve reliability of subject-level activation maps for clinical populations in which group-level analysis may not be feasible or appropriate, for example in a chronic stroke cohort with varying stroke location and degree of tissue damage.
| S-EPMC10370165 | biostudies-literature
Project description:Correlated motion analysis provides a method for understanding communication between and dynamic similarities of biopolymer residues and domains. The typical equal-time correlation matrices-frequently visualized with pseudo-colorings or heat maps-quickly convey large regions of highly correlated motion but hide more subtle similarities of motion. Here we propose a complementary method for visualizing correlations within proteins (or general biopolymers) that quickly conveys intuition about which residues have a similar dynamic behavior. For grouping residues, we use the recently developed non-parametric clustering algorithm HDBSCAN. Although the method we propose here can be used to group residues using correlation as a similarity matrix-the most straightforward and intuitive method-it can also be used to more generally determine groups of residues which have similar dynamic properties. We term these latter groups "Dynamic Domains", as they are based not on spatial closeness but rather closeness in the column space of a correlation matrix. We provide examples of this method across three human proteins of varying size and function-the Nf-Kappa-Beta essential modulator, the clotting promoter Thrombin and the mismatch repair protein (dimer) complex MutS-alpha. Although the examples presented here are from all-atom molecular dynamics simulations, this visualization technique can also be used on correlations matrices built from any ensembles of conformations from experiment or computation.
| S-EPMC5734272 | biostudies-literature
Project description:Speech discrimination that determines whether a participant is speaking at a given moment is essential in investigating human verbal communication. Specifically, in dynamic real-world situations where multiple people participate in, and form, groups in the same space, simultaneous speakers render speech discrimination that is solely based on audio sensing difficult. In this study, we focused on physical activity during speech, and hypothesized that combining audio and physical motion data acquired by wearable sensors can improve speech discrimination. Thus, utterance and physical activity data of students in a university participatory class were recorded, using smartphones worn around their neck. First, we tested the temporal relationship between manually identified utterances and physical motions and confirmed that physical activities in wide-frequency ranges co-occurred with utterances. Second, we trained and tested classifiers for each participant and found a higher performance with the audio-motion classifier (average accuracy 92.2%) than both the audio-only (80.4%) and motion-only (87.8%) classifiers. Finally, we tested inter-individual classification and obtained a higher performance with the audio-motion combined classifier (83.2%) than the audio-only (67.7%) and motion-only (71.9%) classifiers. These results show that audio-motion multimodal sensing using widely available smartphones can provide effective utterance discrimination in dynamic group communications.
| S-EPMC7287755 | biostudies-literature
Project description:Swing in a crew boat, a good jazz riff, a fluid conversation: these tasks require extracting sensory information about how others flow in order to mimic and respond. To determine what factors influence coordination, we build an environment to manipulate incoming sensory information by combining virtual reality and motion capture. We study how people mirror the motion of a human avatar's arm as we occlude the avatar. We efficiently map the transition from successful mirroring to failure using Gaussian process regression. Then, we determine the change in behaviour when we introduce audio cues with a frequency proportional to the speed of the avatar's hand or train individuals with a practice session. Remarkably, audio cues extend the range of successful mirroring to regimes where visual information is sparse. Such cues could facilitate joint coordination when navigating visually occluded environments, improve reaction speed in human-computer interfaces or measure altered physiological states and disease.
| S-EPMC6544895 | biostudies-literature
Project description:In natural audio-visual environments, a change in depth is usually correlated with a change in loudness. In the present study, we investigated whether correlating changes in disparity and loudness would provide a functional advantage in binding disparity and sound amplitude in a visual search paradigm. To test this hypothesis, we used a method similar to that used by van der Burg et al. to show that non-spatial transient (square-wave) modulations of loudness can drastically improve spatial visual search for a correlated luminance modulation. We used dynamic random-dot stereogram displays to produce pure disparity modulations. Target and distractors were small disparity-defined squares (either 6 or 10 in total). Each square moved back and forth in depth in front of the background plane at different phases. The target's depth modulation was synchronized with an amplitude-modulated auditory tone. Visual and auditory modulations were always congruent (both sine-wave or square-wave). In a speeded search task, five observers were asked to identify the target as quickly as possible. Results show a significant improvement in visual search times in the square-wave condition compared to the sine condition, suggesting that transient auditory information can efficiently drive visual search in the disparity domain. In a second experiment, participants performed the same task in the absence of sound and showed a clear set-size effect in both modulation conditions. In a third experiment, we correlated the sound with a distractor instead of the target. This produced longer search times, indicating that the correlation is not easily ignored.
| S-EPMC3355117 | biostudies-literature
Project description:Multisensory signals can enhance the spatial perception of objects and events in the environment. Changes of visual size and auditory intensity provide us with the main cues about motion direction in depth. However, frequency changes in audition and binocular disparity in vision also contribute to the perception of motion in depth. Here, we presented subjects with several combinations of auditory and visual depth-cues to investigate multisensory interactions during processing of motion in depth. The task was to discriminate the direction of auditory motion in depth according to increasing or decreasing intensity. Rising or falling auditory frequency provided an additional within-audition cue that matched or did not match the intensity change (i.e. intensity-frequency (IF) "matched vs. unmatched" conditions). In two-thirds of the trials, a task-irrelevant visual stimulus moved either in the same or opposite direction of the auditory target, leading to audio-visual "congruent vs. incongruent" between-modalities depth-cues. Furthermore, these conditions were presented either with or without binocular disparity. Behavioral data showed that the best performance was observed in the audio-visual congruent condition with IF matched. Brain imaging results revealed maximal response in visual area V3A when all cues provided congruent and reliable depth information (i.e. audio-visual congruent, IF-matched condition including disparity cues). Analyses of effective connectivity revealed increased coupling from auditory cortex to V3A specifically in audio-visual congruent trials. We conclude that within- and between-modalities cues jointly contribute to the processing of motion direction in depth, and that they do so via dynamic changes of connectivity between visual and auditory cortices.
| S-EPMC3838953 | biostudies-literature
Project description:Actors are required to engage in multimodal modulations of their body, face, and voice in order to create a holistic portrayal of a character during performance. We present here the first trimodal analysis, to our knowledge, of the process of character portrayal in professional actors. The actors portrayed a series of stock characters (e.g., king, bully) that were organized according to a predictive scheme based on the two orthogonal personality dimensions of assertiveness and cooperativeness. We used 3D motion capture technology to analyze the relative expansion/contraction of 6 body segments across the head, torso, arms, and hands. We compared this with previous results for these portrayals for 4 segments of facial expression and the vocal parameters of pitch and loudness. The results demonstrated significant cross-modal correlations for character assertiveness (but not cooperativeness), as manifested collectively in a straightening of the head and torso, expansion of the arms and hands, lowering of the jaw, and a rise in vocal pitch and loudness. These results demonstrate what communication theorists refer to as "multichannel reinforcement". We discuss this reinforcement in light of both acting theories and theories of human communication more generally.
| S-EPMC9117657 | biostudies-literature