Dataset Information

Multimodality and the origin of a novel communication system in face-to-face interaction.

ABSTRACT: Face-to-face communication is multimodal at its core: it consists of a combination of vocal and visual signalling. However, current evidence suggests that, in the absence of an established communication system, visual signalling, especially in the form of visible gesture, is a more powerful form of communication than vocalization and therefore likely to have played a primary role in the emergence of human language. This argument is based on experimental evidence of how vocal and visual modalities (i.e. gesture) are employed to communicate about familiar concepts when participants cannot use their existing languages. To investigate this further, we introduce an experiment where pairs of participants performed a referential communication task in which they described unfamiliar stimuli in order to reduce reliance on conventional signals. Visual and auditory stimuli were described in three conditions: using visible gestures only, using non-linguistic vocalizations only and given the option to use both (multimodal communication). The results suggest that even in the absence of conventional signals, gesture is a more powerful mode of communication compared with vocalization, but that there are also advantages to multimodality compared to using gesture alone. Participants with an option to produce multimodal signals had comparable accuracy to those using only gesture, but gained an efficiency advantage. The analysis of the interactions between participants showed that interactants developed novel communication systems for unfamiliar stimuli by deploying different modalities flexibly to suit their needs and by taking advantage of multimodality when required.

SUBMITTER: Macuch Silva V

PROVIDER: S-EPMC7029942 | biostudies-literature | 2020 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Multimodality and the origin of a novel communication system in face-to-face interaction.

Macuch Silva Vinicius V Holler Judith J Ozyurek Asli A Roberts Seán G SG

Royal Society open science 20200115 1

Face-to-face communication is multimodal at its core: it consists of a combination of vocal and visual signalling. However, current evidence suggests that, in the absence of an established communication system, visual signalling, especially in the form of visible gesture, is a more powerful form of communication than vocalization and therefore likely to have played a primary role in the emergence of human language. This argument is based on experimental evidence of how vocal and visual modalitie ...[more]

PMID: 32218922

Similar Datasets

Project description:A successful reciprocal evaluation of social signals serves as a prerequisite for social coherence and empathy. In a previous fMRI study we studied naturalistic communication situations by presenting video clips to our participants and recording their behavioral responses regarding empathy and its components. In two conditions, all three channels transported congruent emotional or neutral information, respectively. Three conditions selectively presented two emotional channels and one neutral channel and were thus bimodally emotional. We reported channel-specific emotional contributions in modality-related areas, elicited by dynamic video clips with varying combinations of emotionality in facial expressions, prosody, and speech content. However, to better understand the underlying mechanisms accompanying a naturalistically displayed human social interaction in some key regions that presumably serve as specific processing hubs for facial expressions, prosody, and speech content, we pursued a reanalysis of the data. Here, we focused on two different descriptions of temporal characteristics within these three modality-related regions [right fusiform gyrus (FFG), left auditory cortex (AC), left angular gyrus (AG) and left dorsomedial prefrontal cortex (dmPFC)]. By means of a finite impulse response (FIR) analysis within each of the three regions we examined the post-stimulus time-courses as a description of the temporal characteristics of the BOLD response during the video clips. Second, effective connectivity between these areas and the left dmPFC was analyzed using dynamic causal modeling (DCM) in order to describe condition-related modulatory influences on the coupling between these regions. The FIR analysis showed initially diminished activation in bimodally emotional conditions but stronger activation than that observed in neutral videos toward the end of the stimuli, possibly by bottom-up processes in order to compensate for a lack of emotional information. The DCM analysis instead showed a pronounced top-down control. Remarkably, all connections from the dmPFC to the three other regions were modulated by the experimental conditions. This observation is in line with the presumed role of the dmPFC in the allocation of attention. In contrary, all incoming connections to the AG were modulated, indicating its key role in integrating multimodal information and supporting comprehension. Notably, the input from the FFG to the AG was enhanced when facial expressions conveyed emotional information. These findings serve as preliminary results in understanding network dynamics in human emotional communication and empathy.

Project description:(1) Background: Teledermatology (TD) has exponentially grown since the onset of COVID-19, as the Face-to-Face Dermatology (F-F/D) modality changed within Public Health Systems. Although studies have been conducted on health results, we did not find any that analyzed the experiences of individuals who received care through TD. Therefore, the main objective of the study was to analyze the experiences of dermatology patients and the communication with health personnel. (2) Methods: A multicenter, controlled, randomized, non-blinded clinical trial was designed. Data were collected during the six months of follow-up. Four-hundred and fifty patients participated in the present study, who were assigned to two different groups: TD and F-F/D. The sociodemographic and clinical characteristics of the participants were collected. The ‘Generic Short Patient Experiences Questionnaire’ (GS-PEQ) was used to assess patients’ experiences, and the Healthcare Communication Questionnaire (HCCQ) was used to measure the communication of patients with healthcare professionals. (3) Results: After six months of follow-up, 450 patients completed the study (TD = 225; F-F/D = 225) of which 53.3% were women, with an average age of 52.16 (SD = 19.97). The main reasons for the consultations were skin lesions (51.56%) located on the head and neck (46.8%), followed by the legs (20.7%). According to the GS-PEQ, TD users indicated having a greater confidence in the professional skills of the doctors (p < 0.01). However, the F-F/D group indicated having received more adequate information about their diagnosis/afflictions (p < 0.01), were more involved in the decisions related to their treatment (p < 0.01), and more satisfied with the help and treatment received (p < 0.01). Regarding the HCCQ, the TD group obtained better assessments with respect to if the patients had been treated in a rude and hasty manner, if the health professionals had addressed them with a smile, and if these could adequately manage the reason for the consultation (p < 0.01). (4) Conclusions: The results of the study generally showed positive experiences and communication. The TD group indicated having received less information about the diagnosis, were less involved in the decisions, and were less satisfied with the help and treatment received. However, they indicated having more confidence on the professional skills of the doctors, and that the work at the institution was better organized. In addition, they perceived better communication skills of the health professionals, although less respect for their privacy.

Dataset Information

Multimodality and the origin of a novel communication system in face-to-face interaction.

Publications

Multimodality and the origin of a novel communication system in face-to-face interaction.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets