Dataset Information

Multi-Modality Emotion Recognition Model with GAT-Based Multi-Head Inter-Modality Attention.

ABSTRACT: Emotion recognition has been gaining attention in recent years due to its applications on artificial agents. To achieve a good performance with this task, much research has been conducted on the multi-modality emotion recognition model for leveraging the different strengths of each modality. However, a research question remains: what exactly is the most appropriate way to fuse the information from different modalities? In this paper, we proposed audio sample augmentation and an emotion-oriented encoder-decoder to improve the performance of emotion recognition and discussed an inter-modality, decision-level fusion method based on a graph attention network (GAT). Compared to the baseline, our model improved the weighted average F1-scores from 64.18 to 68.31% and the weighted average accuracy from 65.25 to 69.88%.

SUBMITTER: Fu C

PROVIDER: S-EPMC7506856 | biostudies-literature | 2020 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Multi-Modality Emotion Recognition Model with GAT-Based Multi-Head Inter-Modality Attention.

Fu Changzeng C Liu Chaoran C Ishi Carlos Toshinori CT Ishiguro Hiroshi H

Sensors (Basel, Switzerland) 20200829 17

Emotion recognition has been gaining attention in recent years due to its applications on artificial agents. To achieve a good performance with this task, much research has been conducted on the multi-modality emotion recognition model for leveraging the different strengths of each modality. However, a research question remains: what exactly is the most appropriate way to fuse the information from different modalities? In this paper, we proposed audio sample augmentation and an emotion-oriented ...[more]

PMID: 32872511

Similar Datasets

Project description:Emotion recognition is key to interpersonal communication and to human-machine interaction. Body expression may contribute to emotion recognition, but most past studies focused on a few motions, limiting accurate recognition. Moreover, emotions in most previous research were acted out, resulting in non-natural motion, which is unapplicable in reality. We present an approach for emotion recognition based on body motion in naturalistic settings, examining authentic emotions, natural movement, and a broad collection of motion parameters. A lab experiment using 24 participants manipulated participants' emotions using pretested movies into five conditions: happiness, relaxation, fear, sadness, and emotionally-neutral. Emotion was manipulated within subjects, with fillers in between and a counterbalanced order. A motion capture system measured posture and motion during standing and walking; a force plate measured center of pressure location. Traditional statistics revealed nonsignificant effects of emotions on most motion parameters; only 7 of 229 parameters demonstrate significant effects. Most significant effects are in parameters representing postural control during standing, which is consistent with past studies. Yet, the few significant effects suggest that it is impossible to recognize emotions based on a single motion parameter. We therefore developed machine learning models to classify emotions using a collection of parameters, and examined six models: k-nearest neighbors, decision tree, logistic regression, and the support vector machine with radial base function and linear and polynomial functions. The decision tree using 25 parameters provided the highest average accuracy (45.8%), more than twice the random guess for five conditions, which advances past studies demonstrating comparable accuracies, due to our naturalistic setting. This research suggests that machine learning models are valuable for emotion recognition in reality and lays the foundation for further progress in emotion recognition models, informing the development of recognition devices (e.g., depth camera), to be used in home-setting human-machine interactions.

Dataset Information

Multi-Modality Emotion Recognition Model with GAT-Based Multi-Head Inter-Modality Attention.

Publications

Multi-Modality Emotion Recognition Model with GAT-Based Multi-Head Inter-Modality Attention.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets